{"id":"68307a46-9aff-4214-a3ce-461febfce593","arxiv_id":"1908.05490","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"This paper introduces MMN, a multivariate score that combines compositionality, statistical association, and substitutability to rank English noun-noun compounds, and reports higher precision and correlation than previous single-signal models.","lead":"Software that reads text needs to know when a phrase means something different from its words, such as 'soap opera'. This paper combines three clues for spotting such phrases and says the combined score works better than older single-clue methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported improvement is not established because the DS_REDDY result is obtained after dropping features based on their observed correlation with the test-set labels, and no held-out evaluation is reported.","rationale":"The reader's weakest_assumption focuses on the normality/independence/right-tail assumption of MMN, and that assumption is indeed contradicted by the DS_REDDY negative correlations and the resulting feature drop. However, the more directly load-bearing flaw for the paper's headline is evaluative: the result that 'remarkably outperforms' is obtained after selecting features on the test-set labels, with no held-out split, no cross-validation, and no model-selection-aware significance testing. The feature-dropping is both a symptom of the failed normality/right-tail assumption and a methodological breach that by itself invalidates the comparison. I therefore partially agree with the reader. The verdict should remain REJECT, since the central empirical claim is not established; a proper held-out evaluation could change the conclusion either way, and the paper as written does not provide it.","tokens_in":11462,"tokens_out":3611,"duration_ms":39254,"concrete_test":"Re-run the DS_REDDY comparison with 10-fold or leave-one-out cross-validation: inside each training fold, decide whether to include NPMI and SDMAm based only on that fold's correlations, then fit MMNs and evaluate Spearman correlation on the held-out fold. If the average held-out MMNs correlation does not exceed INTRCT's 0.433, or if the selected dimensions differ across folds, the reported 0.530 is an artifact of test-set feature selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table 1 (MMNs 0.530 vs. INTRCT 0.433) and Fig. 7. On DS_REDDY, the winning MMNs is not the full multivariate model: after observing that NPMI and SDMAm are negatively correlated with human non-compositionality (Table 1), Sec. 4.2 explicitly drops these two dimensions and evaluates only ADT and INTRCT. This is feature selection on the test-set labels, and with n=90 it can easily overfit the selection criterion; the reported p-values treat the model as fixed and do not account for the post hoc choice. On DS_FARAHMAND, the same set supplies the data for estimating mu and sigma in Eqs. 9-10 and is then ranked and scored for p@k, so the comparison is not a predictive evaluation. The paper itself states that NPMI and SDMAm are not discriminant in this dataset and that the model is 'adjusted' accordingly, which is a self-reported limitation of the evidence. Because the 'remarkably outperforms' conclusion can be explained by this in-sample/post hoc adaptation, the central claim is not supported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multivariate model, MMN (and its smooth variant MMNs), for identifying semantically non-compositional noun-noun compounds. The model combines three types of scores: semantic non-compositionality (using the ADT and INTRCT baselines), statistical association (using association measures such as NPMI and chi-squared), and non-substitutability (using the newly introduced SDMA measures). MMN multiplies rectified deviations of the scores from their means with a term that down-weights high-probability regions under an assumed multivariate normal distribution. The model is evaluated on two datasets, DS_FARAHMAND (with p@k) and DS_REDDY (with Spearman correlation against human scores), where it is reported to outperform the ADT, INTRCT, and MLT baselines. The central claim is that jointly modeling all three characteristics leads to a 'remarkably' better identification of non-compositional phrases.","tokens_in":11676,"tokens_out":4933,"duration_ms":44547,"significance":"If the empirical claims were sound, the paper would make a useful contribution by explicitly combining multiple characteristics of non-compositional phrases in a single score. The proposed model is simple and the code is publicly available, which supports reproducibility. The paper also introduces SDMA as a computationally cheap measure of non-substitutability. However, the current evidence is compromised by in-sample parameter estimation on the evaluation datasets and by post hoc feature selection based on test-set labels, so the significance of the reported improvements is not yet established.","major_comments":[{"comment":"The parameters mu_i and sigma_i^2 of the MMN/MMNs models are estimated by maximum likelihood on DS_FARAHMAND (Eqs. 9-10), and the same dataset is then scored and ranked to compute p@k in Fig. 7. This is an in-sample evaluation, so the reported improvement of MMN/MMNs over ADT, INTRCT, and MLT may be an artifact of fitting the score distribution to the evaluation data. Please re-estimate parameters on a training split (or use cross-validation) and report held-out p@k, with error bars or significance tests.","section":"Sec. 4.1, Eqs. 9-10"},{"comment":"The DS_REDDY comparison is compromised by post hoc feature selection: after observing that NPMI and SDMAm have negative correlations with human non-compositionality scores, Sec. 4.2 states that MLT and MMNs are 'accordingly adjusted' to use only ADT and INTRCT. The reported MMNs correlation of 0.530 therefore evaluates a reduced model whose feature subset was chosen using test-set labels, and with n=90 this selection is highly susceptible to overfitting; the p-values in Table 1 do not account for the selection step. The paper should either pre-specify the feature set or select features on a development portion of the data, and should report the uncertainty of the correlation estimates.","section":"Sec. 4.2, Table 1"},{"comment":"The model's key assumption -- that non-compositional phrases lie to the right of the mean on every dimension and that the constituent scores are near-normal and independent -- is contradicted by the paper's own DS_REDDY results, where NPMI and SDMAm are negatively correlated with human scores (Table 1). The author's response is to drop these dimensions rather than to challenge the model assumption; this limits the generality of the proposed multivariate approach and needs to be addressed, for example by modelling the direction of each dimension explicitly or by justifying why the assumption should hold for new datasets.","section":"Sec. 3.5.1, Eq. 15"},{"comment":"The definitions of pm, ph, and pc in Eqs. 1-3 are only well-defined if the nearest-neighbor sets form disjoint semantic clusters (as the paper notes), but the paper does not describe how such clustering is performed or whether the normalization constant is valid for the fastText neighbor sets actually used. Please clarify the implementation and, if the sum over all word pairs is not exactly 1, how SDMA scores remain comparable across compounds.","section":"Sec. 3.3.3, Eqs. 1-3"}],"minor_comments":[{"comment":"There is a typo in the first paragraph: 'ret tape' should be 'red tape'.","section":"Sec. 1"},{"comment":"The footnote states that the left word is called the head and the right word the modifier, which contradicts the standard right-headed analysis of English noun compounds and is inconsistent with the formulas: pm substitutes the left word (modifier) while keeping the right word (head). Please correct the terminology.","section":"Sec. 3.3.3, footnote 4"},{"comment":"In Eq. 16 the term under the square root is written as '2πσ 2exp'; it should be '2πσ_i^2' to match Eq. 11.","section":"Eq. 16"},{"comment":"The axis label in Fig. 3(b) appears as 'x 10 e18' and 'x 108'; please format the scientific notation correctly.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The conceptual idea is reasonable and the paper is clearly written, but the experimental evaluation is not valid as reported. I would like to see a revised version with proper held-out evaluation and pre-specified features, after which I could recommend acceptance; without that, the central claim lacks support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper builds a multivariate score (MMN) that combines non-compositionality, statistical association, and non-substitutability, and introduces a substitution-driven ratio (SDMA). That combination is genuinely new, and the SDMA idea is worth a look. But the empirical claim that it 'remarkably outperforms' previous models is not supported, because on DS_REDDY the authors drop features after seeing their correlation with the human labels, and on DS_FARAHMAND the MMN parameters are estimated from the same data being ranked.\n\nThe paper does things well. The SDMA formulation is simple and computationally cheap, and the evaluation against association measures and prior non-substitutability models is reasonable for that piece. The idea that non-compositional phrases should sit in the tail of a multivariate distribution is clearly explained, and the model is easy to implement. The GitHub link is supplied.\n\nThe soft spots are serious. On DS_REDDY, Table 1 shows NPMI and SDMAm negatively correlated with human non-compositionality; the paper then 'adjusts' MLT and MMNs to include only ADT and INTRCT. That is feature selection on the test labels, and with n=90 it can easily produce optimistic results. The reported p-values treat the model as fixed, so they don't account for the selection. On DS_FARAHMAND, the same dataset supplies mu and sigma for equations (9)-(10) and then is ranked for p@k; that is an in-sample evaluation. No held-out split, no error bars, no significance testing. The paper itself acknowledges the DS_REDDY issue in-text, which is honest but does not fix the claim.\n\nThe stress-test note is accurate. The central comparison is not a predictive evaluation, so the title-level claim is not established. The model might still work with a proper train/test design, but we don't know from this paper.\n\nWho this is for: people working on MWE detection and collocation extraction will find the SDMA ratio and the multivariate framing interesting, and the paper is readable. It deserves a serious referee, but the referee should ask for a held-out evaluation and a fixed feature set before publication. I would not cite the empirical results as they stand.","headline":"New multivariate score and SDMA ratio are worth knowing, but the reported gains are in-sample and post hoc, so the headline claim is unproven.","tokens_in":12236,"tokens_out":1939,"would_cite":false,"duration_ms":17998,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that non-compositional phrases are best identified as points in the upper tail of several independent score distributions at once, and that a product-of-deviations score built on this idea outperforms single-signal…","keywords":["semantic non-compositionality","multiword expressions","multivariate model","statistical association","non-substitutability","noun compounds","word embeddings","Spearman correlation"],"falsifier":"Compute MMN and MMNs on DS_REDDY with NPMI and SDMAm kept in the model instead of dropped; the paper's own results show those two dimensions are negatively correlated with human non-compositionality scores, so the model's ranking should deteriorate sharply, confirming that the reported 0.530 correlation depends on the dimension-selection step rather than on the multivariate formula alone. Alternatively, fit a full multivariate Gaussian with estimated covariances and compare its Spearman correlation with the independence-based MMNs on both datasets.","tokens_in":11192,"feed_emoji":"🧩","tokens_out":7462,"duration_ms":64522,"temperature":0.7,"pith_summary":"Non-compositional phrases—phrases like soap opera whose meaning does not follow from their parts—are usually detected with a single signal: the gap between a phrase's distributional vector and the composed vectors of its words. This paper claims that the task is better framed as a multivariate problem, because such phrases are exceptional in three ways at once: semantically non-compositional, statistically associated, and non-substitutable. The paper builds a score, MMN (and a smooth variant MMNs), from the product of a phrase's above-mean deviations on all three kinds of scores, weighted by a term that discounts high-probability phrases. On 1,042 English noun compounds, MMN and MMNs beat the best single-signal baselines (ADT, INTRCT) and a multiplicative hybrid (MLT) for almost all values of $k$ in precision-at-$k$; on a second dataset of 90 compounds, MMNs raises Spearman correlation with human non-compositionality judgments from 0.433 to 0.530. The payoff of the multivariate view is that requiring a phrase to be unusual on several independent dimensions at once suppresses compositional phrases that happen to score high on any one cue.","feed_headline":"A multivariate score beats one-cue models on non-compositionality","feed_subtitle":"Combining semantic, statistical, and substitutability signals raises human-judgment correlation from .433 to .530.","key_machinery":"The load-bearing object is the Multivariate Measure of Non-compositionality (MMN), Eq. (12): $\\mathrm{MMN} = \\prod_i r(v_i - \\mu_i)\\,(1 - p(v))$, with $r$ the rectifier $\\max(0, x)$ and $p(v)$ the independence-assumed product of univariate normal densities whose means and variances are estimated by maximum likelihood. The smooth variant MMNs substitutes $\\ln(1 + e^{x})$ for $r(x)$, so dimensions below the mean down-weight rather than veto. The second ingredient is SDMA, which scores non-substitutability as the logarithm of the joint probability of a compound divided by the probability mass of its nearest-neighbour substitution alternatives, computed with fastText word vectors. The machinery works by partitioning the score space into a low-dimensional upper-tail region, and assigning high scores only to phrases that are simultaneously far above the mean on every characteristic while occupying a low-density part of the joint distribution.","core_discovery":"The central claim is that semantic non-compositionality, statistical association, and non-substitutability are complementary measurements of the same underlying phenomenon, and that a model can exploit all three by assuming each score is near-normally distributed and independent. For each candidate phrase, MMN takes the product of rectified deviations $r(v_i - \\mu_i)$ over dimensions, so a phrase below the mean on any dimension gets zero, and multiplies by $1 - p(v)$, where $p(v)$ is the product of the univariate normal densities; MMNs smooths the rectifier to $\\ln(1 + \\exp(v_i - \\mu_i))$, giving partial credit to phrases slightly below a mean. The paper demonstrates on DS_FARAHMAND that this beats ADT (additive composition), INTRCT (interaction regression), and MLT (the multiplicative baseline) across most precision-at-$k$ values, and on DS_REDDY, after dropping the two dimensions that correlate negatively with human scores, MMNs reaches a Spearman correlation of 0.530 with human non-compositionality ratings, up from 0.433 for the best previous baseline. The paper also develops a substitution-driven measure of association (SDMA), a ratio of a compound's joint probability to the summed probability of its nearest-neighbour substitutions, as a practical non-substitutability channel.","pith_inferences":["A direct probabilistic alternative the paper does not test would be to estimate a calibrated posterior $P(\\text{non-compositional} \\mid \\text{scores})$ and compare it with MMN; that comparison would isolate whether the product-of-deviations form or the tail-density discount is the active ingredient.","The same unusual-on-all-axes logic transfers to any multi-signal anomaly detection task, such as flagging terms that are fixed in translation or extracting terminology, where several weak independent signals can be multiplied instead of thresholded.","A testable extension is replacing the rectified-deviation product with a covariance-aware Gaussian: if the paper's independence assumption is the limiting factor, such a model should further improve Spearman correlation on DS_REDDY beyond 0.530."],"forward_implications":["If the multivariate account is right, systems that currently detect non-compositional phrases with a single compositionality score can be improved by adding cheap association and substitution statistics, without retraining a new model.","The SDMA ratio gives a low-cost, wide-coverage proxy for non-substitutability that outperforms earlier log-linear models on DS_FARAHMAND, so it can serve as a reusable feature for other multiword-expression tasks.","Because MMN and MMNs are generic products over dimensions, the same scoring recipe can be applied to any set of characteristic scores, including other syntactic categories such as verb-object constructions.","The DS_REDDY results imply that dimension selection matters: when a general MWE property is not discriminant in a dataset, the model should drop or down-weight that dimension rather than multiply it in."],"supporting_citations":[{"why":"Supplies DS_FARAHMAND, the 1,042-compound dataset with non-compositionality and statistical idiosyncrasy annotations used for the precision-at-k evaluation.","marker":"[Farahmand et al., 2015]"},{"why":"Supplies DS_REDDY, the 90-compound human compositionality dataset, and the additive composition model ADT used as a baseline.","marker":"[Reddy et al., 2011]"},{"why":"Supplies INTRCT, the polynomial-regression non-compositionality baseline that is also a dimension in MLT and MMN.","marker":"[Yazdani et al., 2015]"},{"why":"Adapts ADT to word embeddings, giving the ADT scores used as a baseline and candidate dimension.","marker":"[Salehi et al., 2015]"},{"why":"Supplies NPMI, the normalized statistical-association measure used in MLT and as a candidate dimension in the multivariate model.","marker":"[Bouma, 2009]"},{"why":"Supplies H1 and H2, previous non-substitutability models that SDMA is evaluated against.","marker":"[Farahmand and Henderson, 2016]"},{"why":"Prior nearest-neighbour non-substitutability approach that motivates substitution-based detection of non-compositionality.","marker":"[Kiela and Clark, 2013]"},{"why":"Earlier association-based identification of non-compositional phrases, representing the high-recall family the paper contrasts with.","marker":"[Lin, 1999]"}],"fun_headline_variants":["Multivariate model hits .530 on non-compositionality","Triple-cue model outperforms single-cue for non-compositionality","Combining three signals improves non-compositionality prediction","A multivariate score beats one-cue models on non-compositionality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every retained characteristic score is near-normally distributed, independent of the others, and positively related to non-compositionality, so that a phrase scoring above the mean on all dimensions is exactly the set the model should rank high; if any non-compositional phrase falls below the mean on one dimension, MMN assigns it zero, and if a dimension is negatively related in a particular dataset, the model must drop it by hand.","fun_headline_variants_meta":{"raw":{"variants":["Multivariate model hits .530 on non-compositionality","Triple-cue model outperforms single-cue for non-compositionality","Combining three signals improves non-compositionality prediction","A multivariate score beats one-cue models on non-compositionality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2684,"prompt_tokens":914,"completion_tokens":1770,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1698}},"tokens_in":530,"tokens_out":1770,"duration_ms":12717,"temperature":1.0,"reasoning_tokens":1698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:12:55.634266+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute MMN and MMNs on DS_REDDY with NPMI and SDMAm kept in the model instead of dropped; the paper's own results show those two dimensions are negatively correlated with human non-compositionality scores, so the model's ranking should deteriorate sharply, confirming that the reported 0.530 correlation depends on the dimension-selection step rather than on the multivariate formula alone. Alternatively, fit a full multivariate Gaussian with estimated covariances and compare its Spearman correlation with the independence-based MMNs on both datasets.","supporting_citations":[{"cited_title":"Learning semantic composition to detect non- compositionality of multiword expressions","cited_arxiv_id":null,"evidence_quote":"Supplies DS_FARAHMAND, the 1,042-compound dataset with non-compositionality and statistical idiosyncrasy annotations used for the precision-at-k evaluation."},{"cited_title":"Learning semantic composition to detect non- compositionality of multiword expressions","cited_arxiv_id":null,"evidence_quote":"Supplies INTRCT, the polynomial-regression non-compositionality baseline that is also a dimension in MLT and MMN."}],"review_version":1}