modeling idiosyncratic preferences: how generative...
TRANSCRIPT
Modeling idiosyncratic preferences: How generative knowledge and expressionfrequency jointly determine language structure
Emily Morgan ([email protected])
Roger Levy ([email protected])Department of Linguistics, UC San Diego, 9500 Gilman Drive, La Jolla, CA 92093-0108 USA
Abstract
Most models of choice in language focus on broadly applica-ble generative knowledge, treating item-specific variation asnoise. Focusing on word order preferences in binomial ex-pressions (e.g. bread and butter), we find meaning in theitem-specific variation: more frequent expressions have morepolarized (i.e. frozen) preferences. Of many models consid-ered, only one that takes expression frequency into account canpredict the language-wide distribution of preference strengthsseen in corpus data. Our results support a gradient trade-off inlanguage processing between generative knowledge and item-specific knowledge as a function of frequency.
Keywords: Bayesian modeling; binomial expression; fre-quency; word order
IntroductionA pervasive question in language processing research is howwe reconcile generative knowledge with idiosyncratic prop-erties of specific lexical items. In many cases, the generativeknowledge is the primary object of study, while item-specificidiosyncrasies are treated as noise. For instance, in mod-eling the dative alternation, Bresnan, Cueni, Nikitina, andBaayen (2007) take care to demonstrate that effects of ani-macy, givenness, etc. on structure choice hold even after ac-counting for biases of individual verbs. But the verb biasesthemselves are not subject to any serious investigation. Herewe present evidence that patterns within the item-specificvariation are meaningful, and that by modeling this variation,we not only obtain better models of the phenomenon of in-terest, we also learn more about language structure and itscognitive representation.
Specifically, we will develop a model of word order prefer-ences for binomial expressions of the form X and Y (i.e. breadand butter preferred over butter and bread). Binomial order-ing preferences are in part determined by generative knowl-edge of violable constraints which reference the semantic,phonological, and lexical properties of the constituent words(e.g. short-before-long; Cooper & Ross, 1975; McDonald,Bock, & Kelly, 1993), but speakers also have idiosyncraticpreferences for known expressions, particularly those withhigh frequency (Morgan & Levy, 2015; Siyanova-Chanturia,Conklin, & van Heuven, 2011). Here we will demonstratethat explicitly modeling these idiosyncrasies both produces abetter predictive model for novel expressions and also con-strains our theory of these expressions’ cognitive representa-tions.
Specifically, we identify two reasons why such a model isadvantageous:
1. Models identify both rules and exceptions.One intrinsic reason that modeling idiosyncrasies is advan-tageous is because identifying exceptions can help identifyrules. In a traditional linguistic setting (e.g. identifying rulesfor past tense formation), we rely upon intuition to deter-mine what is the grammatical rule and which verbs shouldbe treated as exceptions. In the case of binomial expressions,we likewise expect there to be exceptions to the rules. Forexample, there is in general a strong constraint to put menbefore women; however, ladies and gentlemen is preferredover the reverse due to its conventionalized formal use. Butcompared with past tense formation, the rules that determinebinomial ordering are far more complex and gradient, suchthat using traditional linguistic analysis to determine the fullset of rules is not viable. In this case, we require our modelnot only to identify what the rules are but simultaneously todetermine which expressions must be treated as exceptions.Having such a model is useful for empirical cognitive sci-ence, e.g. for disentangling the effects of people’s generativeknowledge from effects of their item-specific linguistic expe-rience on language processing (Morgan & Levy, 2015).
2. Models relate cognitive representations tolanguage-wide structure.As a further benefit, models can help us understand howstructural properties of the language relate to people’s cog-nitive linguistic representations. In particular, let us look atthe distribution of preferences for binomial expressions takenfrom a subset of the Google Books corpus (described later inCreating the Corpus.) Each binomial can be assigned a pref-erence strength corresponding to how frequently it appears inalphabetical order, from 0 (always in non-alphabetical order)to 0.5 (perfectly balanced) to 1 (always alphabetical). Bino-mials which always or nearly always appear in one order aresaid to be frozen. The distribution of preference strengths isshown in Figure 1. Preferences have a multimodal distribu-tion with modes at the extremes as well as around 0.5. Thisdistribution poses a challenge to standard models of binomialpreferences. As we will show later, standard models predictonly a single mode around 0.5. In other words, the true distri-bution of binomial expressions includes more frozen binomi-als than standard models predict. As we develop a model thataccounts for this multimodal distribution, we will see that thislanguage-structural fact puts constraints on our theories of in-dividuals’ cognitive representations of binomial expressions.
In the remainder of this paper, we first describe how wedeveloped a new corpus of binomial expressions. We then
explore a variety of models with differing levels of ability tomodel item-specific idiosyncrasies. Finally, we return to theissue of how these models inform us about cognitive repre-sentations of language.
Histogram of binomial types
Proportion of occurrences in alphabetical order
Density
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.5
1.0
1.5
Figure 1: Binomial preferences are multimodally distributedin corpus data
Creating the CorpusWe extracted all Noun-and-Noun binomials from theparsed section of the Brown corpus (Marcus, Santorini,Marcinkiewicz, & Taylor, 1999) using the following Tregex(Levy & Galen, 2006) search pattern:/ˆN/=top < (/ˆNN/ !$, (/,/ > =top) .((CC <: and > =top) . (/ˆNN/ > =top)))
This pattern finds all Noun-and-Noun sequences dominatedby a Noun Phrase which are not preceded by a comma (toexclude the final pair in lists of more than two elements), atotal of 1280 tokens.
Binomials were coded for a variety of constraints, origi-nally described by Benor and Levy (2006) but restricted tothe subset determined to be most relevant for predicting or-dering preferences by Morgan and Levy (2015):
Length The shorter word (in syllables) comes first, e.g.abused and neglected.No final stress The final syllable of the second word shouldnot be stressed, e.g. abused and neglected.Lapse Avoid unstressed syllables in a row, e.g. FARMS andHAY-fields vs HAY-fields and FARMSFrequency The more frequent word comes first, e.g. brideand groom.Formal markedness The word with more general meaningor broader distribution comes first, e.g. boards and two-by-fours.Perceptual markedness Elements that are more closelyconnected to the speaker come first. This constraint encom-passes Cooper and Ross’s (1975) ‘Me First’ constraint and in-cludes numerous subconstraints, e.g.: animates precede inan-imates; concrete words precede abstract words; e.g. deer andtrees.Power The more powerful or culturally prioritized wordcomes first, e.g. clergymen and parishioners.Iconic/scalar sequencing Elements that exist in sequenceshould be ordered in sequence, e.g. achieved and maintained.
Cultural Centrality The more culturally central or commonelement should come first, e.g. oranges and grapefruits.Intensity The element with more intensity appears first, e.g.war and peace.Pragmatic Catch-all constraint for direct influences on or-dering from the local context. In particular, an element thathas been previously mentioned should come first.
The metrical constraints, Length and No final stress, were au-tomatically extracted from the CMU Pronouncing Dictionary(2014), augmented by manual annotations when necessary.Word frequency was taken from the Google Books corpus,counting occurrences from 1900 or later (Lin et al., 2012).Semantic constraints were hand coded by two independentcoders (drawing from the first author and two trained researchassistants). Discrepancies were resolved through discussion.
For each binomial, we obtained the number of occurrencesin both possible orders in the Google Books n-grams corpusfrom 1900 or later. Items containing proper names, thosewith errors in the given parses, those for which the Pragmaticconstraint applied, and those with less than 1000 total occur-rences across both orders were excluded from analysis, leav-ing 594 binomial expression types.
ModelsWe will develop four models of binomial ordering prefer-ences: a standard logistic regression, a mixed-effects logis-tic regression, and two hierarchical Bayesian beta-binomialmodels. All are based on the idea of using logistic regres-sion to combine the constraints described above in a weightedfashion to produce an initial preference estimate for each bi-nomial. The models differ in whether and how they explic-itly model the fact that true preferences will be distributed id-iosyncratically around these estimates. The standard logisticregression includes no explicit representation of item-specificidiosyncrasies. The mixed-effect logistic regression includesrandom intercepts which account for item-specific idiosyn-crasies, but which are constrained to be distributed normallyaround the initial prediction. The two Bayesian models as-sume that item-specific preferences are drawn from a betadistribution whose mean is determined by the initial predic-tion. In the first of these models, the concentration of the betadistribution is fixed, while in the second, it varies with thefrequency of the binomial in question.
Evaluation
One obvious criterion for evaluating a model is how well itpredicts known binomial preferences (i.e. the corpus data).For this, we report R2(X , X̂) as well as mean L1 error,1N ΣN
i=1 |x̂i− xi|, where x̂i is the model prediction for how of-ten binomial i occurs in a given order, and xi is the true corpusproportion.
In addition to considering model predictions for each in-dividual item, we want to consider the overall distribution ofpreferences within the language. As we will see, a model can
provide good predictions for individual items without cor-rectly capturing the language-wide multimodal distributionof these expressions’ preference strengths. Thus our seconddesideratum will be the shape of the histogram of expressionpreferences.
Logistic regression
Logistic regression is the standard for modeling syntactic al-ternations, both for binomial expressions specifically (e.g.Benor & Levy, 2006; Morgan & Levy, 2015) as well othersyntactic alternations (e.g. Bresnan et al., 2007; Jaeger,2010).Thus we begin by constructing a baseline logistic re-gression model. Benor and Levy have argued that one shouldtrain such a model on binomial types rather than binomial to-kens because otherwise a large number of tokens for a smallnumber of overrepresented types can skew the results. Whileagreeing with this logic, we note that to train only a singleinstance of each type is to ignore a vast amount of data aboutthe gradient nature of binomial preferences. As a compro-mise, we instead train a model on binomial tokens, using to-ken counts from the Google Books corpus, with each tokenweighted in inverse proportion to how many tokens there arefor that binomial type, i.e. a type with 1000 tokens will haveeach token weighted at 1/1000. In this way, we preserve thegradient information about ordering preferences (via the di-versity of outcomes among tokens) while still weighting eachtype equally. The constraints listed above are used as pre-dictors. Outcomes are coded as whether or not the binomialtoken is in alphabetical order.
For this and all future models, predictions are generated forall training items using 20-fold cross validation. Results forall models can be seen in Figure 2. While the logistic regres-sion does model a reasonable job of predicting preferencesfor individual items, it does not capture the multimodal dis-tribution of preference strengths seen in the corpus data. Weproceed to consider models in which item-specific idiosyn-crasies are modeled explicitly.
Mixed-effects regression
By far the most common method in language modeling foraccounting for item-specific idiosyncrasies is mixed-effectsregression models (Jaeger, 2008). Formally, this model as-sumes that idiosyncratic preferences are distributed normally(in logit space) around the point estimate given by the fixed-effects components of the regression model.
We train a mixed-effect logistic regression on binomial to-kens using the lme4 package in R. We use as predictors thesame fixed effects as before, plus a random intercept for bino-mial types. As described above, the fitted model now predictsa distribution, rather than a single point estimate, for a novelbinomial. To make predictions for our (cross-validated) noveldata, we sampled 1000 times from this distribution for eachitem. The histogram in Figure 2(c) shows the full sample dis-tribution across all items. In order to generate point estimatepredictions for computing L1 and R2 (shown in Figure 2(b)),
we take the sample median for each item, which optimizesthe L1 error.
Including random intercepts improves neither our point es-timates nor our language-wide distribution prediction. Appar-ently, the normal distribution of the random intercepts is notwell suited to capturing the true distribution of binomial pref-erences. In particular, for a given item, the normality of ran-dom effects in logit space leads to predictions that are skewedtowards the extremities of probability space.1
Hierarchical Bayesian beta-binomial model
Having seen that normally distributed random intercepts donot adequately capture the distribution of item-specific pref-erences, we introduce the beta distribution as a potentiallybetter way to model this distribution. The beta distribution,defined on the interval [0,1], has two parameters: one whichdetermines the mean of the draws from the distribution, andone which determines the concentration, i.e. whether drawsare likely to be clustered around the mean versus distributedtowards 0 and 1. For example, for a beta distribution witha mean of 0.7, a high concentration implies that most drawswill be close to 0.7, while a low concentration implies thatroughly 70% of draws will be close to 1 and 30% of drawswill be close to 0. When we treat the output of the beta dis-tribution as a predicted binomial preference, a high concen-tration corresponds to a pressure to maintain variation whilea low concentration corresponds to a pressure to regularize.
In order to incorporate the beta distribution into our modelof binomial preferences, we combine the logistic regressionand the beta distribution in a hierarchical Bayesian model(Gelman et al., 2013), as shown in Figure 3. For each item,the model determines a mean µ via standard logistic regres-sion, using the same predictors as before. The model alsofits a concentration parameter ν. These two parameters deter-mine a beta distribution from which the binomial preference π
is draw. Observed data is drawn from a Binomial distributionwith parameter π.
We fit this model using the rjags package in R (Plum-mer, 2003). After a burn-in period of 2000 iterations, werun for 2000 more iterations sampling every 20 iterations. Inorder to predict novel data, we fix the point estimates for theregression coefficients β̂ and the concentration parameter ν.We then sample 1000 draws of π for each item. As withthe mixed-effects model, the histogram in Figure 2(c) showsthe full sample distribution, while point estimates (the samplemedian) are used to calculate L1 error and R2 (Figure 2(b)).
This model performs better on L1 and R2 than the mixed-effects model, but still worse than the initial logistic regres-sion. The predicted histogram shows hints of the multimodaldistribution seen in corpus data, but is overall too flat.
1An alternative method of prediction for novel items would be totake the median random intercept in logit space, i.e. to set all randomintercepts to 0. This method yields results that are very similar to—but all-around slightly worse than—the original regression model.
θβ
β̂ X̂ θν
µ ν
π
D
Mn
N
N: # unordered binomial typesMn: Frequency of binomial nX̂ : Predictors (i.e. generativeconstraints)β̂: Regression coefficientsθ: Uninformative priors
µ =1
1+ e−X̂ ·β̂ν∼ exp(θν)π∼ Beta(µ,ν)D∼ Binomial(π,Mn)
Figure 3: Our initial hierarchical Bayesian beta-binomialmodel. The set of nodes culiminating in µ implements a stan-dard linear regression. The output of this regression deter-mines the mean of the beta distribution (with ν determiningthe concentration) from which π and finally the observed dataitself is drawn.
Beta-binomial with a variable concentrationparameterA crucial fact that we have not taken into account in previousmodels is the role of frequent reuse in shaping the distributionof expressions within the language. In particular, the degreeto which an expression takes on a polarized preference maydepend upon its frequency. For our final model, we buildupon the beta-binomial model in the previous section by pa-rameterizing the concentration parameter by the frequency ofthe (unordered) binomial expression:
ν = exp(α+β · log(Mn)) (1)
where Mn is the total number of occurrences of binomial n inboth orders.
Training and testing of the model are identical to above.We find that β = −0.26 is significantly different from 0(t99 = −94; p < 2.2e− 16), indicating that the concentrationparameter changes significantly as a function of frequency:less frequent expressions have more dense distributions whilemore frequent expressions have more polarized distributions.The concentration parameter as a function of expression fre-quency is visualized in Figure 5. We find that this model gen-erates the best predictions of all our models as measured byboth L1 error and R2. Moreover, this model has correctly pre-dicted the multimodal distribution of expression preferences.
DiscussionOverall, we found that all models made approximately simi-larly good best-guess predictions for binomials they weren’ttrained on, but the frequency-sensitive beta-binomial modelwas clearly superior in predicting the language-wide distribu-tion of idiosyncratic binomial-specific ordering preferences.
θβ
β̂ X̂ Mn
θα,β
µ ν
α,β
π
D
Mn
N
Figure 4: Hierarchical Bayesian beta-binomial model withvariable concentration parameter
-18 -16 -14 -12
02
46
8
Log frequency
Spa
rsity
par
amet
er ν
-18 -16 -14 -12
02
46
8
Log frequency
Spa
rsity
par
amet
er ν
Figure 5: Concentration parameter ν as a function of fre-quency with 95% confidence intervals. (Left) Parameteriza-tion given in Eq. 1. (Right) Alternate parameterization withcubic splines, for comparison.
This model also indicates that more frequent binomials areon average more polarized.
This modeling finding supports Morgan and Levy (2015)’sclaim that generative knowledge and item-specific direct ex-perience trade off gradiently in language processing, such thatprocessing of novel or infrequent items relies upon generativeknowledge, with reliance upon item-specific experience in-creasing with increasing frequency of exposure. Morgan andLevy support this claim with behavioral data, showing thatempirical preferences for binomials which are completelynovel depend on generative constraints while preferences forfrequent expressions depend primarily on frequency of expe-rience with each order. Our modeling results augment this ar-gument by demonstrating that this trade-off is likewise neces-sary in order to predict the language-wide distribution of pref-erence strengths. In particular, we can conceive of generativeknowledge as providing a prior for ordering preferences. Un-der our final model, the logistic regression component servesan estimate of generative knowledge, which generates pref-erences clustered unimodally around 0.5. The amount of di-rect experience one has with an expression then modulateswhether it conforms to this prior or whether it deviates: Itemswith low frequency have a high concentration: they maintaintheir variability and continue to contribute to the mode around
0.5. Items with high frequency have a low concentration: theyare more likely to regularize and contribute to the modes at 0and 1. Crucially, the inclusion of expression frequency as apredictor of the concentration of the beta distribution is nec-essary in order to achieve this effect in the model, demon-strating that expressions are indeed relying differentially ongenerative knowledge versus direct experience depending ontheir frequency.
This finding fits with previous models of cultural transmis-sion in which, in general, preferences gravitate towards theprior (Griffiths & Kalish, 2005), but with sufficient expo-sure, exceptions can be learned (e.g. irregular verbs; Lieber-man, Michel, Jackson, Tang, & Nowak, 2007). However,this raises a question which is not answered by our or oth-ers’ models: why don’t all expressions converge to their priorpreferences eventually? We present two possibilities.
One possibility is that people’s probabilistic transmissionbehavior differs at different frequencies. Convergence to theprior relies upon probability matching: people must repro-duce variants in approximately the proportion in which theyhave encountered them. However, this is not the only possi-ble behavior. Another possibility is that people preferentiallyreproduce the most frequent variant they have encountered, tothe exclusion of all other variants, a process known as regu-larizing. If people’s tendency to probability match versus reg-ularize is dependent on the frequency of expression in ques-tion (with more regularizing at high frequencies), this couldproduce the pattern of more polarized expressions at higherfrequencies that we see in our data. Another possibility is thatthere is some other unspecified exogenous source of pressuretowards regularization, as for instance seems to be the case inchild language acquisition (Hudson Kam & Newport, 2009).This pressure might be weak enough that it is overwhelmedby convergence towards the prior at lower frequencies, butcan be maintained for items with high enough frequencies tohave sufficient exposure to deviate from the prior. Furtherwork is necessary to disentangle these explanations.
In addition to contributing to our understanding of bino-mial expression processing, we have demonstrated the valueof modeling the distribution of idiosyncratic preferences intwo ways. First, it has improved our ability to predict pref-erences for novel items, by better differentiating the rule-following training data from the exceptions. Second, thismodel turns an observation about language-wide structure(the multimodal distribution of preferences) into a constrainton our theory of the cognitive representation and processingof language (more polarization at higher frequencies).
ReferencesBenor, S., & Levy, R. (2006). The Chicken or the Egg?
A Probabilistic Analysis of English Binomials. Lan-guage, 82(2), 233–278.
Bresnan, J., Cueni, A., Nikitina, T., & Baayen, R. H. (2007).Predicting the dative alternation. Cognitive foundationsof interpretation.
The CMU Pronouncing Dictionary . (2014).Cooper, W. E., & Ross, J. R. (1975). World Order. In
R. E. Grossman, L. J. San, & T. J. Vance (Eds.), Pa-pers from the parasession on functionalism (pp. 63–111). Chicago: Chicago Linguistics Society.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Ve-htari, A., & Rubin, D. B. (2013). Bayesian Data Anal-ysis, Third Edition. CRC Press.
Griffiths, T. L., & Kalish, M. L. (2005). A Bayesian view oflanguage evolution by iterated learning. Proceedingsof the 27th annual conference of the Cognitive ScienceSociety, 827–832.
Hudson Kam, C. L., & Newport, E. L. (2009). Getting it rightby getting it wrong: When learners change languages.Cognitive Psychology, 59(1), 30–66.
Jaeger, T. F. (2008). Categorical data analysis: Away fromANOVAs (transformation or not) and towards logitmixed models. Journal of Memory and Language,59(4), 434–446.
Jaeger, T. F. (2010). Redundancy and reduction: Speakersmanage syntactic information density. Cognitive Psy-chology, 61(1), 23–62.
Levy, R., & Galen, A. (2006). Tregex and Tsurgeon. 5thInternational Conference on Language Resources andEvaluation (LREC).
Lieberman, E., Michel, J.-B., Jackson, J., Tang, T., & Nowak,M. A. (2007). Quantifying the evolutionary dynamicsof language. Nature, 449(7163), 713–716.
Lin, Y., Michel, J.-B., Aiden, E. L., Orwant, J., Brockman,W., & Petrov, S. (2012). Syntactic Annotations for theGoogle Books Ngram Corpus. Proceedings of the 50thAnnual Meeting of the Association for ComputationalLinguistics, 169–174.
Marcus, M., Santorini, B., Marcinkiewicz, M. A., & Taylor,A. (1999). Treebank-3 LDC99T42. Linguistic DataConsortium. Philadelphia.
McDonald, J., Bock, K., & Kelly, M. (1993). Word andworld order: Semantic, phonological, and metrical de-terminants of serial position. Cognitive Psychology, 25,188–230.
Morgan, E., & Levy, R. (2015). Abstract knowledge ver-sus direct experience in processing of binomial expres-sions. Manuscript submitted for publication.
Plummer, M. (2003). JAGS: A program for analysis ofBayesian graphical models using Gibbs sampling.
Siyanova-Chanturia, A., Conklin, K., & van Heuven, W. J. B.(2011). Seeing a phrase “time and again” matters: Therole of phrasal frequency in the processing of multi-word sequences. Journal of Experimental Psychology:Learning, Memory, and Cognition, 37(3), 776–784.
Logisticregression
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Intense
Culture
Icon
Power
Percept
Form
Freq
Lapse
Stress
Length
Intercept
0 1Parameter estimate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Corpus proportion
Mod
el p
ropo
rtion
L1= 0.169 (0.006)R2 = 0.368 (0.020)
Density
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.5
1.0
1.5
2.0
2.5
Proportion of occurrencesin alphabetical order
Mixed-effectsregressionwith randomintercept
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Intense
Culture
Icon
Power
Percept
Form
Freq
Lapse
Stress
Length
Intercept
−0.5 0.0 0.5 1.0 1.5 2.0 2.5Parameter estimate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Corpus proportionM
odel
pro
porti
onL1= 0.173 (0.006)R2 = 0.355 (0.014)
Density
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.4
0.8
1.2
Proportion of occurrencesin alphabetical order
Beta-binomialmodel
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Intense
Culture
Icon
Power
Percept
Form
Freq
Lapse
Stress
Length
Intercept
0.0 0.5 1.0 1.5Parameter estimate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Corpus proportion
Mod
el p
ropo
rtion
L1= 0.172 (0.006)R2 = 0.358 (0.016)
Density
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Proportion of occurrencesin alphabetical order
Beta-binomialmodel withvariable con-centration
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
Intense
Culture
Icon
Power
Percept
Form
Freq
Lapse
Stress
Length
Intercept
0.0 0.5 1.0 1.5Parameter estimate
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Corpus proportion
Mod
el p
ropo
rtion
L1= 0.168 (0.006)R2 = 0.373 (0.015)
Density
0.0 0.2 0.4 0.6 0.8 1.0
0.00.20.40.60.81.01.2
Proportion of occurrencesin alphabetical order
(a) (b) (c)
Figure 2: For each of our four models, we display: (a) Parameter estimates for the logistic regression component. Dots showpoint estimates with bars indicating standard errors. (b) Predictions for each item, as well as mean by-type L1 error and R2 withby-fold standard errors. (c) Language-wide predicted distribution of preference strengths.