REVIEW 4 major objections 4 minor 13 references
A Multivariate Model for Representing Semantic Non-compositionality
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that non-compositional phrases are best identified as points in the upper tail of several independent score distributions at once, and that a product-of-deviations score built on this idea outperforms single-signal…
desk verdict New multivariate score and SDMA ratio are worth knowing, but the reported gains are in-sample and post hoc, so the headline claim is unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Multivariate Measure of Non-compositionality (MMN), Eq. (12): $\mathrm{MMN} = \prod_i r(v_i - \mu_i)\,(1 - p(v))$, with $r$ the rectifier $\max(0, x)$ and $p(v)$ the independence-assumed product of univariate normal densities whose means and variances are estimated by maximum likelihood. The smooth variant MMNs substitutes $\ln(1 + e^{x})$ for $r(x)$, so dimensions below the mean down-weight rather than veto. The second ingredient is SDMA, which scores non-substitutability as the logarithm of the joint probability of a compound divided by the probability mass of its nearest-neighbour substitution alternatives, computed with fastText word vectors. The machinery works by partitioning the score space into a low-dimensional upper-tail region, and assigning high scores only to phrases that are simultaneously far above the mean on every characteristic while occupying a low-density part of the joint distribution.
What would settle it
Compute MMN and MMNs on DS_REDDY with NPMI and SDMAm kept in the model instead of dropped; the paper's own results show those two dimensions are negatively correlated with human non-compositionality scores, so the model's ranking should deteriorate sharply, confirming that the reported 0.530 correlation depends on the dimension-selection step rather than on the multivariate formula alone. Alternatively, fit a full multivariate Gaussian with estimated covariances and compare its Spearman correlation with the independence-based MMNs on both datasets.
Extended reading notes
Core claim
The central claim is that semantic non-compositionality, statistical association, and non-substitutability are complementary measurements of the same underlying phenomenon, and that a model can exploit all three by assuming each score is near-normally distributed and independent. For each candidate phrase, MMN takes the product of rectified deviations $r(v_i - \mu_i)$ over dimensions, so a phrase below the mean on any dimension gets zero, and multiplies by $1 - p(v)$, where $p(v)$ is the product of the univariate normal densities; MMNs smooths the rectifier to $\ln(1 + \exp(v_i - \mu_i))$, giving partial credit to phrases slightly below a mean. The paper demonstrates on DS_FARAHMAND that this beats ADT (additive composition), INTRCT (interaction regression), and MLT (the multiplicative baseline) across most precision-at-$k$ values, and on DS_REDDY, after dropping the two dimensions that correlate negatively with human scores, MMNs reaches a Spearman correlation of 0.530 with human non-compositionality ratings, up from 0.433 for the best previous baseline. The paper also develops a substitution-driven measure of association (SDMA), a ratio of a compound's joint probability to the summed probability of its nearest-neighbour substitutions, as a practical non-substitutability channel.
Load-bearing premise
The load-bearing premise is that every retained characteristic score is near-normally distributed, independent of the others, and positively related to non-compositionality, so that a phrase scoring above the mean on all dimensions is exactly the set the model should rank high; if any non-compositional phrase falls below the mean on one dimension, MMN assigns it zero, and if a dimension is negatively related in a particular dataset, the model must drop it by hand.
Editorial extensions
If this is right
- If the multivariate account is right, systems that currently detect non-compositional phrases with a single compositionality score can be improved by adding cheap association and substitution statistics, without retraining a new model.
- The SDMA ratio gives a low-cost, wide-coverage proxy for non-substitutability that outperforms earlier log-linear models on DS_FARAHMAND, so it can serve as a reusable feature for other multiword-expression tasks.
- Because MMN and MMNs are generic products over dimensions, the same scoring recipe can be applied to any set of characteristic scores, including other syntactic categories such as verb-object constructions.
- The DS_REDDY results imply that dimension selection matters: when a general MWE property is not discriminant in a dataset, the model should drop or down-weight that dimension rather than multiply it in.
Reading between the lines
- A direct probabilistic alternative the paper does not test would be to estimate a calibrated posterior $P(\text{non-compositional} \mid \text{scores})$ and compare it with MMN; that comparison would isolate whether the product-of-deviations form or the tail-density discount is the active ingredient.
- The same unusual-on-all-axes logic transfers to any multi-signal anomaly detection task, such as flagging terms that are fixed in translation or extracting terminology, where several weak independent signals can be multiplied instead of thresholded.
- A testable extension is replacing the rectified-deviation product with a covariance-aware Gaussian: if the paper's independence assumption is the limiting factor, such a model should further improve Spearman correlation on DS_REDDY beyond 0.530.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a multivariate model, MMN (and its smooth variant MMNs), for identifying semantically non-compositional noun-noun compounds. The model combines three types of scores: semantic non-compositionality (using the ADT and INTRCT baselines), statistical association (using association measures such as NPMI and chi-squared), and non-substitutability (using the newly introduced SDMA measures). MMN multiplies rectified deviations of the scores from their means with a term that down-weights high-probability regions under an assumed multivariate normal distribution. The model is evaluated on two datasets, DS_FARAHMAND (with p@k) and DS_REDDY (with Spearman correlation against human scores), where it is reported to outperform the ADT, INTRCT, and MLT baselines. The central claim is that jointly modeling all three characteristics leads to a 'remarkably' better identification of non-compositional phrases.
Significance. If the empirical claims were sound, the paper would make a useful contribution by explicitly combining multiple characteristics of non-compositional phrases in a single score. The proposed model is simple and the code is publicly available, which supports reproducibility. The paper also introduces SDMA as a computationally cheap measure of non-substitutability. However, the current evidence is compromised by in-sample parameter estimation on the evaluation datasets and by post hoc feature selection based on test-set labels, so the significance of the reported improvements is not yet established.
major comments (4)
- [Sec. 4.1, Eqs. 9-10] The parameters mu_i and sigma_i^2 of the MMN/MMNs models are estimated by maximum likelihood on DS_FARAHMAND (Eqs. 9-10), and the same dataset is then scored and ranked to compute p@k in Fig. 7. This is an in-sample evaluation, so the reported improvement of MMN/MMNs over ADT, INTRCT, and MLT may be an artifact of fitting the score distribution to the evaluation data. Please re-estimate parameters on a training split (or use cross-validation) and report held-out p@k, with error bars or significance tests.
- [Sec. 4.2, Table 1] The DS_REDDY comparison is compromised by post hoc feature selection: after observing that NPMI and SDMAm have negative correlations with human non-compositionality scores, Sec. 4.2 states that MLT and MMNs are 'accordingly adjusted' to use only ADT and INTRCT. The reported MMNs correlation of 0.530 therefore evaluates a reduced model whose feature subset was chosen using test-set labels, and with n=90 this selection is highly susceptible to overfitting; the p-values in Table 1 do not account for the selection step. The paper should either pre-specify the feature set or select features on a development portion of the data, and should report the uncertainty of the correlation estimates.
- [Sec. 3.5.1, Eq. 15] The model's key assumption -- that non-compositional phrases lie to the right of the mean on every dimension and that the constituent scores are near-normal and independent -- is contradicted by the paper's own DS_REDDY results, where NPMI and SDMAm are negatively correlated with human scores (Table 1). The author's response is to drop these dimensions rather than to challenge the model assumption; this limits the generality of the proposed multivariate approach and needs to be addressed, for example by modelling the direction of each dimension explicitly or by justifying why the assumption should hold for new datasets.
- [Sec. 3.3.3, Eqs. 1-3] The definitions of pm, ph, and pc in Eqs. 1-3 are only well-defined if the nearest-neighbor sets form disjoint semantic clusters (as the paper notes), but the paper does not describe how such clustering is performed or whether the normalization constant is valid for the fastText neighbor sets actually used. Please clarify the implementation and, if the sum over all word pairs is not exactly 1, how SDMA scores remain comparable across compounds.
minor comments (4)
- [Sec. 1] There is a typo in the first paragraph: 'ret tape' should be 'red tape'.
- [Sec. 3.3.3, footnote 4] The footnote states that the left word is called the head and the right word the modifier, which contradicts the standard right-headed analysis of English noun compounds and is inconsistent with the formulas: pm substitutes the left word (modifier) while keeping the right word (head). Please correct the terminology.
- [Eq. 16] In Eq. 16 the term under the square root is written as '2πσ 2exp'; it should be '2πσ_i^2' to match Eq. 11.
- [Fig. 3] The axis label in Fig. 3(b) appears as 'x 10 e18' and 'x 108'; please format the scientific notation correctly.
Circularity Check
The main performance claim depends on in-sample parameter estimation and test-label-based feature selection.
-
fitted input called prediction
[Section 3.5.1 (Eqs. 9-10, 15) and Section 4.1, Fig. 7]
"The parametersµ1..µn andσ2 1..σ2 n of this multivariate distribution can be estimated by maximum likelihood estimation overm phrases as follows: µi = 1 m Pm j=1 vj i (9) ... The performance of different models for different values of k are shown in Fig. 7. The multivariate models MMN and MMNs outperform MLT for most values of k while they outperform ADT and INTRCT for almost all values of k by a relatively large margin."
MMN's parameters (Eqs. 9-10) are the per-dimension means and variances computed over the same DS_FARAHMAND phrases whose p@k ranking is then reported in Fig. 7. No held-out split is described, so the reported 'outperformance' measures how well the model can rank the very items used to estimate its own mean/variance parameters. This is an in-sample fit, not a predictive evaluation: the ranking is statistically forced to favor items that lie far from this same set's mean and variance, which explains part of the gain over baselines that do not use such set-derived statistics.
-
fitted input called prediction
[Section 4.2, Table 1]
"In theory, high NPMI and SDMAm is a signal for non-compositionality, however, as discussed above, this is not the case in this particular dataset. Hence, we accordingly adjust MLT and MMNs to incorporate only the scores with positive correlations with human scores i.e, ADT and INTRCT."
The decision to drop NPMI and SDMAm is made after observing their negative Spearman correlations with the human non-compositionality labels in DS_REDDY (Table 1). The model is then re-scored and evaluated against those same labels, yielding MMNs ρ = 0.530. Because the feature set is selected using the evaluation labels, the reported correlation and p-values treat the model as fixed and do not account for this post hoc selection; the claimed improvement over fixed baselines is therefore not a valid predictive comparison.
full rationale
The paper's central claim that MMN/MMNs 'remarkably outperforms' previous models rests on two evaluation choices that reduce the result to the evaluation data by construction. On DS_FARAHMAND, the multivariate distribution parameters (µ and σ², Eqs. 9-10) are estimated by maximum likelihood from the same 1042 compounds that are later ranked and scored for p@k; no train/test separation is reported, so Fig. 7 measures in-sample fit rather than identification performance on unseen candidates. On DS_REDDY, the model configuration itself is selected after inspecting the target-label correlations: Sec. 4.2 explicitly drops NPMI and SDMAm because they are negatively correlated with the human scores, then evaluates the adjusted model on those same scores. That post hoc feature selection invalidates the reported p-values and makes the 0.433-to-0.530 improvement a fitted result rather than a fixed-model prediction. The reliance on the author's own dataset (DS_FARAHMAND, Farahmand et al. 2015) and on INTRCT (Yazdani et al. 2015, co-authored by the present author) adds context but is secondary; the core circularity is the in-sample fitting and label-guided feature selection. The multivariate formulation itself is a novel combination, and the SDMA measures are independently defined, so a score of 6 (partial circularity) rather than 8 or 10 is appropriate.
Assumptions & free parameters
free parameters (3)
- k, number of nearest neighbors for substitutes =
5
- L, smoothing constant in SDMA =
unspecified
- Per-dimension means and variances mu_i, sigma_i^2 for MMN =
not reported; estimated on the evaluation set
assumptions (4)
- domain assumption Each characteristic score vi is near-normally distributed and independent.
- domain assumption Non-compositional phrases are denser to the right of the mean on every characteristic.
- ad hoc to paper Eq 1-3 form a true probability distribution only if words are hard-clustered into disjoint semantic groups.
- domain assumption fastText 300d Wikipedia vectors provide valid semantic substitutes via k-nearest-neighbors.
Cite this review
Pith. "Pith review of A Multivariate Model for Representing Semantic Non-compositionality." pith.science (2026). https://pith.science/paper/UFSL2CJ5
@misc{pith2026190805490,
author = {Pith},
title = {Pith review of: A Multivariate Model for Representing Semantic Non-compositionality},
year = {2026},
howpublished = {\url{https://pith.science/paper/UFSL2CJ5}},
note = {Machine review of arXiv:1908.05490}
}
read the original abstract
Semantically non-compositional phrases constitute an intriguing research topic in Natural Language Processing. Semantic non-compositionality --the situation when the meaning of a phrase cannot be derived from the meaning of its components, is the main characteristic of such phrases, however, they bear other characteristics such as high statistical association and non-substitutability. In this work, we present a model for identifying non-compositional phrases that takes into account all of these characteristics. We show that the presented model remarkably outperforms the existing models of identifying non-compositional phrases that mostly focus only on one of these characteristics.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[12]
doi: 10.18653/v1/W16-1809. V . Laparra, G. Camps-Valls, and J. Malo. Iterative gaussianization: From ica to random rotations.IEEE Transactions on Neural Networks, 22(4):537–549, April
-
[13]
ISSN 1045-9227. doi: 10.1109/TNN.2011.2106511. 13
-
[1990]
G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In From F orm to Meaning: Processing Texts Automatically, Proceedings of the Biennial GSCL Conference 2009, volume Normalized, pages 31–40, Tübingen,
work page 2009
-
[1998]
Association for Computational Linguistics. doi: 10.3115/980691.980779. URL http://dx.doi.org/10.3115/980691.980779. Timothy Baldwin and Su Nam Kim. Multiword expressions. In Handbook of Natural Language Processing, second edition., pages 267–292. CRC Press,
-
[1999]
Association for Computational Linguistics. ISBN 1-55860-609-3. doi: 10.3115/1034678.1034730. URL http://dx.doi.org/10.3115/1034678.1034730. Pasi Tapanainen, Jussi Piitulainen, and Timo Järvinen. Idiomatic object usage and support verbs. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Confer...
-
[2001]
A generic framework for multiword expressions treatment: from acquisition to applications
Carlos Ramisch. A generic framework for multiword expressions treatment: from acquisition to applications. In Proceedings of ACL 2012 Student Research Workshop , pages 61–66. Association for Computational Linguistics,
work page 2012
-
[2005]
Association for Computational Linguistics. doi: 10.3115/1220575.1220688. URL http://dx.doi.org/10.3115/1220575.1220688. Graham Katz and Eugenie Giesbrecht. Automatic identification of non-compositional multi-word expressions using latent semantic analysis. In Proceedings of the Workshop on Multiword Expressions: Identifying and Exploiting Underlying Proper...
-
[2006]
Association for Computational Linguistics. ISBN 1-932432-84-1. URL http://dl.acm.org/citation.cfm?id=1613692.1613696. Karl Moritz Hermann, Phil Blunsom, and Stephen Pulman. An unsupervised ranking model for noun-noun composi- tionality. In Proceedings of the First Joint Conference on Lexical and Computational Semantics , pages 132–141. Association for Com...
Show all 13 references
-
[2010]
An empirical model of multiword expression decomposability
Timothy Baldwin, Colin Bannard, Takaaki Tanaka, and Dominic Widdows. An empirical model of multiword expression decomposability. In Proceedings of the ACL 2003 workshop on Multiword expressions: analysis, acquisition and treatment-V olume 18, pages 89–96. Association for Compu...
2003
-
[2011]
Is knowledge-free induction of multiword unit dictionary headwords a solved problem
Patrick Schone and Daniel Jurafsky. Is knowledge-free induction of multiword unit dictionary headwords a solved problem. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing , pages 100–108,
2001
-
[2013]
Learning semantic composition to detect non- compositionality of multiword expressions
Majid Yazdani, Meghdad Farahmand, and James Henderson. Learning semantic composition to detect non- compositionality of multiword expressions. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1733–1742, Lisbon, Portugal, September
2015
-
[2015]
URL http://aclweb.org/anthology/D15-1201
Association for Computational Linguistics. URL http://aclweb.org/anthology/D15-1201. Silvio Cordeiro, Carlos Ramisch, Marco Idiart, and Aline Villavicencio. Predicting the compositionality of nominal compounds: Giving word embeddings a hard time. In Proceedings of the 54th Ann...
1986
-
[2016]
URL http://aclweb.org/anthology/P16-1187
doi: 10.18653/v1/P16-1187. URL http://aclweb.org/anthology/P16-1187. Meghdad Farahmand, Aaron Smith, and Joakim Nivre. A multiword expression data set: Annotating non- compositionality and conventionalization for english noun compounds. In Proceedings of the 11th Workshop on M...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.