Pith. sign in

REVIEW 4 major objections 4 minor 13 references

A Multivariate Model for Representing Semantic Non-compositionality

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read The paper claims that non-compositional phrases are best identified as points in the upper tail of several independent score distributions at once, and that a product-of-deviations score built on this idea outperforms single-signal…

desk verdict New multivariate score and SDMA ratio are worth knowing, but the reported gains are in-sample and post hoc, so the headline claim is unproven. read the letter →

arxiv 1908.05490 v1 pith:UFSL2CJ5 submitted 2019-08-15 cs.CL cs.AI

classification cs.CLcs.AI
keywords semanticnon-compositionalitymultiwordexpressionsmultivariatemodelstatisticalassociationnon-substitutabilitynouncompoundswordembeddingsSpearmancorrelation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Non-compositional phrases—phrases like soap opera whose meaning does not follow from their parts—are usually detected with a single signal: the gap between a phrase's distributional vector and the composed vectors of its words. This paper claims that the task is better framed as a multivariate problem, because such phrases are exceptional in three ways at once: semantically non-compositional, statistically associated, and non-substitutable. The paper builds a score, MMN (and a smooth variant MMNs), from the product of a phrase's above-mean deviations on all three kinds of scores, weighted by a term that discounts high-probability phrases. On 1,042 English noun compounds, MMN and MMNs beat the best single-signal baselines (ADT, INTRCT) and a multiplicative hybrid (MLT) for almost all values of $k$ in precision-at-$k$; on a second dataset of 90 compounds, MMNs raises Spearman correlation with human non-compositionality judgments from 0.433 to 0.530. The payoff of the multivariate view is that requiring a phrase to be unusual on several independent dimensions at once suppresses compositional phrases that happen to score high on any one cue.

What carries the argument

The load-bearing object is the Multivariate Measure of Non-compositionality (MMN), Eq. (12): $\mathrm{MMN} = \prod_i r(v_i - \mu_i)\,(1 - p(v))$, with $r$ the rectifier $\max(0, x)$ and $p(v)$ the independence-assumed product of univariate normal densities whose means and variances are estimated by maximum likelihood. The smooth variant MMNs substitutes $\ln(1 + e^{x})$ for $r(x)$, so dimensions below the mean down-weight rather than veto. The second ingredient is SDMA, which scores non-substitutability as the logarithm of the joint probability of a compound divided by the probability mass of its nearest-neighbour substitution alternatives, computed with fastText word vectors. The machinery works by partitioning the score space into a low-dimensional upper-tail region, and assigning high scores only to phrases that are simultaneously far above the mean on every characteristic while occupying a low-density part of the joint distribution.

What would settle it

Compute MMN and MMNs on DS_REDDY with NPMI and SDMAm kept in the model instead of dropped; the paper's own results show those two dimensions are negatively correlated with human non-compositionality scores, so the model's ranking should deteriorate sharply, confirming that the reported 0.530 correlation depends on the dimension-selection step rather than on the multivariate formula alone. Alternatively, fit a full multivariate Gaussian with estimated covariances and compare its Spearman correlation with the independence-based MMNs on both datasets.

Watch

Extended reading notes

Core claim

The central claim is that semantic non-compositionality, statistical association, and non-substitutability are complementary measurements of the same underlying phenomenon, and that a model can exploit all three by assuming each score is near-normally distributed and independent. For each candidate phrase, MMN takes the product of rectified deviations $r(v_i - \mu_i)$ over dimensions, so a phrase below the mean on any dimension gets zero, and multiplies by $1 - p(v)$, where $p(v)$ is the product of the univariate normal densities; MMNs smooths the rectifier to $\ln(1 + \exp(v_i - \mu_i))$, giving partial credit to phrases slightly below a mean. The paper demonstrates on DS_FARAHMAND that this beats ADT (additive composition), INTRCT (interaction regression), and MLT (the multiplicative baseline) across most precision-at-$k$ values, and on DS_REDDY, after dropping the two dimensions that correlate negatively with human scores, MMNs reaches a Spearman correlation of 0.530 with human non-compositionality ratings, up from 0.433 for the best previous baseline. The paper also develops a substitution-driven measure of association (SDMA), a ratio of a compound's joint probability to the summed probability of its nearest-neighbour substitutions, as a practical non-substitutability channel.

Load-bearing premise

The load-bearing premise is that every retained characteristic score is near-normally distributed, independent of the others, and positively related to non-compositionality, so that a phrase scoring above the mean on all dimensions is exactly the set the model should rank high; if any non-compositional phrase falls below the mean on one dimension, MMN assigns it zero, and if a dimension is negatively related in a particular dataset, the model must drop it by hand.

Editorial extensions

If this is right

  • If the multivariate account is right, systems that currently detect non-compositional phrases with a single compositionality score can be improved by adding cheap association and substitution statistics, without retraining a new model.
  • The SDMA ratio gives a low-cost, wide-coverage proxy for non-substitutability that outperforms earlier log-linear models on DS_FARAHMAND, so it can serve as a reusable feature for other multiword-expression tasks.
  • Because MMN and MMNs are generic products over dimensions, the same scoring recipe can be applied to any set of characteristic scores, including other syntactic categories such as verb-object constructions.
  • The DS_REDDY results imply that dimension selection matters: when a general MWE property is not discriminant in a dataset, the model should drop or down-weight that dimension rather than multiply it in.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct probabilistic alternative the paper does not test would be to estimate a calibrated posterior $P(\text{non-compositional} \mid \text{scores})$ and compare it with MMN; that comparison would isolate whether the product-of-deviations form or the tail-density discount is the active ingredient.
  • The same unusual-on-all-axes logic transfers to any multi-signal anomaly detection task, such as flagging terms that are fixed in translation or extracting terminology, where several weak independent signals can be multiplied instead of thresholded.
  • A testable extension is replacing the rectified-deviation product with a covariance-aware Gaussian: if the paper's independence assumption is the limiting factor, such a model should further improve Spearman correlation on DS_REDDY beyond 0.530.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a multivariate model, MMN (and its smooth variant MMNs), for identifying semantically non-compositional noun-noun compounds. The model combines three types of scores: semantic non-compositionality (using the ADT and INTRCT baselines), statistical association (using association measures such as NPMI and chi-squared), and non-substitutability (using the newly introduced SDMA measures). MMN multiplies rectified deviations of the scores from their means with a term that down-weights high-probability regions under an assumed multivariate normal distribution. The model is evaluated on two datasets, DS_FARAHMAND (with p@k) and DS_REDDY (with Spearman correlation against human scores), where it is reported to outperform the ADT, INTRCT, and MLT baselines. The central claim is that jointly modeling all three characteristics leads to a 'remarkably' better identification of non-compositional phrases.

Significance. If the empirical claims were sound, the paper would make a useful contribution by explicitly combining multiple characteristics of non-compositional phrases in a single score. The proposed model is simple and the code is publicly available, which supports reproducibility. The paper also introduces SDMA as a computationally cheap measure of non-substitutability. However, the current evidence is compromised by in-sample parameter estimation on the evaluation datasets and by post hoc feature selection based on test-set labels, so the significance of the reported improvements is not yet established.

major comments (4)
  1. [Sec. 4.1, Eqs. 9-10] The parameters mu_i and sigma_i^2 of the MMN/MMNs models are estimated by maximum likelihood on DS_FARAHMAND (Eqs. 9-10), and the same dataset is then scored and ranked to compute p@k in Fig. 7. This is an in-sample evaluation, so the reported improvement of MMN/MMNs over ADT, INTRCT, and MLT may be an artifact of fitting the score distribution to the evaluation data. Please re-estimate parameters on a training split (or use cross-validation) and report held-out p@k, with error bars or significance tests.
  2. [Sec. 4.2, Table 1] The DS_REDDY comparison is compromised by post hoc feature selection: after observing that NPMI and SDMAm have negative correlations with human non-compositionality scores, Sec. 4.2 states that MLT and MMNs are 'accordingly adjusted' to use only ADT and INTRCT. The reported MMNs correlation of 0.530 therefore evaluates a reduced model whose feature subset was chosen using test-set labels, and with n=90 this selection is highly susceptible to overfitting; the p-values in Table 1 do not account for the selection step. The paper should either pre-specify the feature set or select features on a development portion of the data, and should report the uncertainty of the correlation estimates.
  3. [Sec. 3.5.1, Eq. 15] The model's key assumption -- that non-compositional phrases lie to the right of the mean on every dimension and that the constituent scores are near-normal and independent -- is contradicted by the paper's own DS_REDDY results, where NPMI and SDMAm are negatively correlated with human scores (Table 1). The author's response is to drop these dimensions rather than to challenge the model assumption; this limits the generality of the proposed multivariate approach and needs to be addressed, for example by modelling the direction of each dimension explicitly or by justifying why the assumption should hold for new datasets.
  4. [Sec. 3.3.3, Eqs. 1-3] The definitions of pm, ph, and pc in Eqs. 1-3 are only well-defined if the nearest-neighbor sets form disjoint semantic clusters (as the paper notes), but the paper does not describe how such clustering is performed or whether the normalization constant is valid for the fastText neighbor sets actually used. Please clarify the implementation and, if the sum over all word pairs is not exactly 1, how SDMA scores remain comparable across compounds.
minor comments (4)
  1. [Sec. 1] There is a typo in the first paragraph: 'ret tape' should be 'red tape'.
  2. [Sec. 3.3.3, footnote 4] The footnote states that the left word is called the head and the right word the modifier, which contradicts the standard right-headed analysis of English noun compounds and is inconsistent with the formulas: pm substitutes the left word (modifier) while keeping the right word (head). Please correct the terminology.
  3. [Eq. 16] In Eq. 16 the term under the square root is written as '2πσ 2exp'; it should be '2πσ_i^2' to match Eq. 11.
  4. [Fig. 3] The axis label in Fig. 3(b) appears as 'x 10 e18' and 'x 108'; please format the scientific notation correctly.

Circularity Check

2 steps flagged · score 6.0 of 10

The main performance claim depends on in-sample parameter estimation and test-label-based feature selection.

  1. fitted input called prediction [Section 3.5.1 (Eqs. 9-10, 15) and Section 4.1, Fig. 7]
    "The parametersµ1..µn andσ2 1..σ2 n of this multivariate distribution can be estimated by maximum likelihood estimation overm phrases as follows: µi = 1 m Pm j=1 vj i (9) ... The performance of different models for different values of k are shown in Fig. 7. The multivariate models MMN and MMNs outperform MLT for most values of k while they outperform ADT and INTRCT for almost all values of k by a relatively large margin."

    MMN's parameters (Eqs. 9-10) are the per-dimension means and variances computed over the same DS_FARAHMAND phrases whose p@k ranking is then reported in Fig. 7. No held-out split is described, so the reported 'outperformance' measures how well the model can rank the very items used to estimate its own mean/variance parameters. This is an in-sample fit, not a predictive evaluation: the ranking is statistically forced to favor items that lie far from this same set's mean and variance, which explains part of the gain over baselines that do not use such set-derived statistics.

  2. fitted input called prediction [Section 4.2, Table 1]
    "In theory, high NPMI and SDMAm is a signal for non-compositionality, however, as discussed above, this is not the case in this particular dataset. Hence, we accordingly adjust MLT and MMNs to incorporate only the scores with positive correlations with human scores i.e, ADT and INTRCT."

    The decision to drop NPMI and SDMAm is made after observing their negative Spearman correlations with the human non-compositionality labels in DS_REDDY (Table 1). The model is then re-scored and evaluated against those same labels, yielding MMNs ρ = 0.530. Because the feature set is selected using the evaluation labels, the reported correlation and p-values treat the model as fixed and do not account for this post hoc selection; the claimed improvement over fixed baselines is therefore not a valid predictive comparison.

full rationale

The paper's central claim that MMN/MMNs 'remarkably outperforms' previous models rests on two evaluation choices that reduce the result to the evaluation data by construction. On DS_FARAHMAND, the multivariate distribution parameters (µ and σ², Eqs. 9-10) are estimated by maximum likelihood from the same 1042 compounds that are later ranked and scored for p@k; no train/test separation is reported, so Fig. 7 measures in-sample fit rather than identification performance on unseen candidates. On DS_REDDY, the model configuration itself is selected after inspecting the target-label correlations: Sec. 4.2 explicitly drops NPMI and SDMAm because they are negatively correlated with the human scores, then evaluates the adjusted model on those same scores. That post hoc feature selection invalidates the reported p-values and makes the 0.433-to-0.530 improvement a fitted result rather than a fixed-model prediction. The reliance on the author's own dataset (DS_FARAHMAND, Farahmand et al. 2015) and on INTRCT (Yazdani et al. 2015, co-authored by the present author) adds context but is secondary; the core circularity is the in-sample fitting and label-guided feature selection. The multivariate formulation itself is a novel combination, and the SDMA measures are independently defined, so a score of 6 (partial circularity) rather than 8 or 10 is appropriate.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central model adds a new score SDMA and a new combination MMN, but it draws on fitted distribution parameters, an undefined smoothing constant, and a normality-tail assumption that the paper's own second dataset violates.

free parameters (3)
  • k, number of nearest neighbors for substitutes = 5
    Set after trying k=3 and k=7; used in Eq 1-3 for pm, ph, pc and hence in all SDMA scores.
  • L, smoothing constant in SDMA = unspecified
    Eq 1-3 add one to the numerator and write N+L in the denominator; L is never defined, yet it changes the alternative probabilities.
  • Per-dimension means and variances mu_i, sigma_i^2 for MMN = not reported; estimated on the evaluation set
    Eq 9-10 estimate these from the same dataset on which p@k is measured, making the ranking in-sample.
assumptions (4)
  • domain assumption Each characteristic score vi is near-normally distributed and independent.
    Sec 3.5.1 Eq 8-11; Jarque-Bera tests show departures, and chi2 is acknowledged to be non-normal.
  • domain assumption Non-compositional phrases are denser to the right of the mean on every characteristic.
    Sec 3.5 opening; DS_REDDY contradicts this for NPMI and SDMAm, acknowledged in Sec 4.2.
  • ad hoc to paper Eq 1-3 form a true probability distribution only if words are hard-clustered into disjoint semantic groups.
    The paper states this requirement but does not implement hard clustering; SDMA still uses the sums.
  • domain assumption fastText 300d Wikipedia vectors provide valid semantic substitutes via k-nearest-neighbors.
    Used throughout Sec 3.3.3 to generate alternatives; no validation that these are synonyms in the relevant sense.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Multivariate Model for Representing Semantic Non-compositionality." pith.science (2026). https://pith.science/paper/UFSL2CJ5

@misc{pith2026190805490,
  author       = {Pith},
  title        = {Pith review of: A Multivariate Model for Representing Semantic Non-compositionality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UFSL2CJ5}},
  note         = {Machine review of arXiv:1908.05490}
}
read the original abstract

Semantically non-compositional phrases constitute an intriguing research topic in Natural Language Processing. Semantic non-compositionality --the situation when the meaning of a phrase cannot be derived from the meaning of its components, is the main characteristic of such phrases, however, they bear other characteristics such as high statistical association and non-substitutability. In this work, we present a model for identifying non-compositional phrases that takes into account all of these characteristics. We show that the presented model remarkably outperforms the existing models of identifying non-compositional phrases that mostly focus only on one of these characteristics.

Figures

Figures reproduced from arXiv: 1908.05490 by the authors.

Figure 1
Figure 1. Performance of AMs in terms of p@ k on DS_FARAHMAND. Let us first define the probability of alternatives as the probability of alternative compounds for the compound w1w2. We define this probability in three different ways which we refer to as pm (alternatives generated by substituting the modifier4 ), ph (alternatives generated by substituting the head) and pc (alternatives generated substituting the head and the m… view at source ↗
Figure 2
Figure 2. Performance of SDMAs in terms of p@ k in comparison with AMs, H1, and H2 on DS_FARAHMAND. 3.4 Hybrid Baseline for Identifying Non-compositionality In the previous section, we introduced various measures to identify general and specific characteristics of non￾compositional phrases. In particular, we presented AMs to identify statistical association, SDMAs to identify non￾substitutability, and ADT and INTRCT to identi… view at source ↗
Figure 3
Figure 3. Distributions of NPMI and χ 2 in DS_FARAHMAND. The distribution of INTRCT on this dataset can be seen in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Distributions of SDMAs in DS_FARAHMAND. 5 10 15 20 25 30 35 40 45 0 50 100 150 200 250 All Non−compositional (a) INTRCT 1.5 2 2.5 3 3.5 4 0 20 40 60 80 100 120 140 160 180 All Non−compositional (b) log − INTRCT [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: a: Distribution of INTRCT in DS_FARAHMAND. b: Distribution of log − INTRCT in DS_FARAHMAND. assumption between vi 7 , a multivariate probability for d can be estimated with respect to the values of its various characteristics as follows:8 p(v) = p(v1)p(v2)..p(vn) = p(v…
Figure 6
Figure 6. Figure 6: A multivariate normal distribution with hypothetical variables [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: We saw that ADT and INTRCT are two strong baselines from previous work that identify non-compositional phrases based only on their specific characteristic i.e. non-compositionality. MLT improves on the performance of these baselines implying the advantages of consideri…
Figure 8
Figure 8. Figure 8: Distribution of non-compositionality (inverse human compositionality) scores in [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Gaussianized distribution of non-compositionality (inverse human compositionality) scores in [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Distribution of characteristic and human scores in [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

  1. [12]

    doi: 10.18653/v1/W16-1809. V . Laparra, G. Camps-Valls, and J. Malo. Iterative gaussianization: From ica to random rotations.IEEE Transactions on Neural Networks, 22(4):537–549, April

  2. [13]

    doi: 10.1109/TNN.2011.2106511

    ISSN 1045-9227. doi: 10.1109/TNN.2011.2106511. 13

  3. [1990]

    G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In From F orm to Meaning: Processing Texts Automatically, Proceedings of the Biennial GSCL Conference 2009, volume Normalized, pages 31–40, Tübingen,

  4. [1998]

    doi: 10.3115/980691.980779

    Association for Computational Linguistics. doi: 10.3115/980691.980779. URL http://dx.doi.org/10.3115/980691.980779. Timothy Baldwin and Su Nam Kim. Multiword expressions. In Handbook of Natural Language Processing, second edition., pages 267–292. CRC Press,

  5. [1999]

    ISBN 1-55860-609-3

    Association for Computational Linguistics. ISBN 1-55860-609-3. doi: 10.3115/1034678.1034730. URL http://dx.doi.org/10.3115/1034678.1034730. Pasi Tapanainen, Jussi Piitulainen, and Timo Järvinen. Idiomatic object usage and support verbs. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Confer...

  6. [2001]

    A generic framework for multiword expressions treatment: from acquisition to applications

    Carlos Ramisch. A generic framework for multiword expressions treatment: from acquisition to applications. In Proceedings of ACL 2012 Student Research Workshop , pages 61–66. Association for Computational Linguistics,

  7. [2005]

    doi: 10.3115/1220575.1220688

    Association for Computational Linguistics. doi: 10.3115/1220575.1220688. URL http://dx.doi.org/10.3115/1220575.1220688. Graham Katz and Eugenie Giesbrecht. Automatic identification of non-compositional multi-word expressions using latent semantic analysis. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Proper...

  8. [2006]

    ISBN 1-932432-84-1

    Association for Computational Linguistics. ISBN 1-932432-84-1. URL http://dl.acm.org/citation.cfm?id=1613692.1613696. Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. An unsupervised ranking model for noun-noun composi- tionality. In Proceedings of the First Joint Conference on Lexical and Computational Semantics , pages 132–141. Association for Com...

Show all 13 references
  1. [2010]

    An empirical model of multiword expression decomposability

    Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment-V olume 18, pages 89–96. Association for Compu...

  2. [2011]

    Is knowledge-free induction of multiword unit dictionary headwords a solved problem

    Patrick Schone and Daniel Jurafsky. Is knowledge-free induction of multiword unit dictionary headwords a solved problem. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing , pages 100–108,

  3. [2013]

    Learning semantic composition to detect non- compositionality of multiword expressions

    Majid Yazdani, Meghdad Farahmand, and James Henderson. Learning semantic composition to detect non- compositionality of multiword expressions. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1733–1742, Lisbon, Portugal, September

  4. [2015]

    URL http://aclweb.org/anthology/D15-1201

    Association for Computational Linguistics. URL http://aclweb.org/anthology/D15-1201. Silvio Cordeiro, Carlos Ramisch, Marco Idiart, and Aline Villavicencio. Predicting the compositionality of nominal compounds: Giving word embeddings a hard time. In Proceedings of the 54th Ann...

  5. [2016]

    URL http://aclweb.org/anthology/P16-1187

    doi: 10.18653/v1/P16-1187. URL http://aclweb.org/anthology/P16-1187. Meghdad Farahmand, Aaron Smith, and Joakim Nivre. A multiword expression data set: Annotating non- compositionality and conventionalization for english noun compounds. In Proceedings of the 11th Workshop on M...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.