Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics

Alexandre Salle; Aline Villavicencio

REVIEW 2 major objections 4 minor 25 references

Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In low-rank factorization of PMI matrices, positive PMI carries nearly all semantic information, while negative PMI contributes almost exclusively syntactic information.

desk verdict A useful probe of PMI weighting variants under LexVec, but the central semantic/syntactic dissociation is confounded by the asymmetric use of negative sampling. read the letter →

arxiv 1908.06941 v1 pith:PFXKH3AC submitted 2019-08-19 cs.CL

Alexandre Salle , Aline Villavicencio This is my paper · ORCID

classification cs.CL

keywords pointwisemutualinformationpositivePMInegativePPMIwordembeddingsmatrixfactorizationdistributionalsemanticssyntactic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Pointwise mutual information (PMI) is the standard re-weighting for word co-occurrence counts, but its negative values are usually clipped to zero because they include many unreliable counts from finite corpora. This paper asks what that clipping actually loses. Using a low-rank factorization of PMI matrices and probing the resulting embeddings on semantic and syntactic tasks, it argues that positive PMI carries nearly all semantic information and most syntactic information, while negative PMI carries almost exclusively syntactic information. If the claim holds, the common PPMI shortcut is not just a numerical convenience; it separates meaning from grammar, and negative PMI becomes an available unsupervised signal for syntax.

What carries the argument

The machinery is the family of clipped PMI matrices $CPMI_z(w,c) = \max(z, PMI(w,c))$, where $z=0$ gives PPMI and a small negative threshold such as $z=-2$ captures most of the negative spectrum, together with a sign-restricted stochastic gradient training scheme that skips updates for target cells whose value has the wrong sign. That scheme produces separate positive-only and negative-only models from the same factorization framework, so differences on downstream probes can be attributed to the information carried by each sign. A supplementary variant, NNEGPMI, normalizes only the negative branch of PMI, testing whether scale in the negative spectrum matters.

What would settle it

Train a negative-PMI-only model with enough extra epochs or negative-cell sampling that it receives the same number of gradient updates as the positive-PMI-only model, and check whether its semantic-task scores remain near chance; if they rise substantially, the original sign-asymmetry conclusion is confounded by update count.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sign of PMI splits distributional information by type. In a low-rank factorization trained with window sampling and negative sampling, a model that only sees non-negative PMI values performs as well as or better than full-spectrum models on nearly every semantic task and on most syntactic probes; a model that only sees negative PMI values performs near chance on semantic tasks but approaches full-model accuracy on part-of-speech tagging and syntactic sentence probes from the same architecture. The authors conclude that positive PMI encodes both semantics and syntax, negative PMI encodes almost exclusively syntax, and that clipping the negative spectrum to zero, as PPMI does, discards little of what matters for meaning.

Load-bearing premise

The experiments assume that skipping stochastic gradient updates for cells whose PMI has the wrong sign isolates the information carried by the other sign, but because positive cells appear far more often in the training stream than negative cells, the sign-specific models receive very different numbers of updates, so the observed differences may partly reflect training quantity rather than information content.

Editorial extensions

If this is right

PPMI embeddings are sufficient when the goal is semantic similarity or textual meaning, since the clipped negative spectrum contributes little to those tasks.
Negative PMI can be treated as an unsupervised syntax channel, useful for tasks that need word-order, agreement, or part-of-speech information without labeled syntactic data.
Scale information in the negative spectrum is not needed: clipping negatives at $-2$ or normalizing only the negative branch preserves most performance, while normalizing the full PMI spectrum hurts.
The advantage of full-spectrum models on rare-word similarity and analogies suggests negative PMI should be exploited selectively for rare words rather than applied globally.
Because positive PMI alone matches full models on most tasks, embedding pipelines can skip negative-PMI computation and reduce memory and update cost without losing semantic content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct testable extension is a rare-word hybrid: add clipped negative PMI only for low-frequency words, predicting it beats both pure PPMI and full-spectrum models by giving rare words repulsion without contaminating common-word semantics.
If negative PMI really encodes syntax, then PPMI-only embeddings should show characteristic blind spots on syntactic probing tasks, such as near-chance scores on agreement or word-order probes, while mixed models do not.
The paper's connection to indirect negative evidence suggests a cross-linguistic prediction: languages with richer inflectional morphology should show stronger syntactic signal in negative PMI than analytic languages, because agreement constraints create more systematic non-cooccurrence patterns.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper asks a good question — what negative PMI actually contributes in low-rank factorization of PMI matrices — and it runs a wide battery of probes (SimLex, RW, analogies, STSB, SentEval, POS) to answer it. The headline finding, that +PMI alone captures most semantics and syntax while -PMI-only models look almost purely syntactic, is plausible and would be useful if it held. The paper also introduces two clean variants, CPMI_z and NNEGPMI, and ships code, which is credit-worthy.

But the central comparison is confounded, and the stress-test note is right about the sharper version of the problem. LexVec's training stream is not just the cooccurrence-weighted cells; it contains k negative-sampling updates per center word. In the +PPMI condition, every non-positive target is skipped, so all negative-sampling updates (target 0) are dropped. In the -CPMI-2 condition, non-positive targets are kept, so the model receives the full negative-sampling stream with clipped target -2. The two conditions therefore differ in the presence of negative sampling as an optimization component, not merely in the sign of the PMI signal. The paper's own Table 1 shows the size of this effect: +PPMI scores 56.1 on GSem and 50.3 on GSyn, while full PPMI — which only adds the zero-target negative-sampling updates — jumps to 80.3 and 62.2. Attributing that jump to "negative PMI information" is not supported, since the target is 0 in those updates. Negative sampling is a known regularizer and rare-word improver; the "syntactic" behavior of -CPMI-2 may well be that optimizer artifact, not evidence about matrix content.

The reader's update-count concern is related, but Figure 1 suggests the sampled stream is not wildly unbalanced by sign; the asymmetric role of negative sampling is the more decisive problem.

What survives is narrower: within LexVec with its default optimization, clipping negative PMI at 0 (PPMI) is just as good as more elaborate treatments, and full NPMI hurts. The paper does not establish the semantic/syntactic dissociation. A clean control would be to factor the matrix with SVD (no negative sampling) or to add negative-sampling updates with target 0 to the +PPMI condition. Absent that, I'd treat the main claim as unproven.

Who is this for? Researchers working on embedding objectives and association measures. It deserves a serious referee, but the referee should ask for the control. I'd send it to review, with the expectation of heavy revision.

Recommendation: engage with it, but only after the confound is addressed. The question is worth getting right.

Referee Report

2 major / 4 minor

Summary. The manuscript studies the roles of positive and negative pointwise mutual information in low-rank factorization of PMI-weighted co-occurrence matrices. It proposes two PMI variants, CPMI_z (clipping PMI from below at a value z, recovering PPMI for z=0) and NNEGPMI (normalizing only the negative part of PMI), and evaluates them with the LexVec factorization model. To separate the contributions of the two signs, the authors train LexVec with SGD updates skipped when the target cell value is non-positive (+PPMI condition) or positive (-CPMI-2/-NNEGPMI conditions). On semantic and syntactic benchmarks, +PPMI performs well on both semantic and syntactic tasks, whereas the negative-only models perform near baseline on semantics but close to full models on POS tagging and the SentEval syntactic probing tasks. The paper concludes that most semantics and syntax come from +PMI, that -PMI carries almost exclusively syntactic information, and that PPMI is a well-grounded choice because full-spectrum models perform no better.

Significance. The question is relevant for embedding design and for the theory of distributional semantics. If the main empirical claim were established, the finding would be practically useful: it would justify discarding negative PMI for semantic tasks and might allow negative PMI to be treated as a syntactic signal. The paper is a clearly written empirical study with a broad task battery (word similarity, analogies, sentence similarity, probing tasks, POS tagging), introduces two transparent PMI variants, and releases code. The main empirical claim is falsifiable, and the full-versus-ablated comparisons in Table 1 are easy to reproduce. At present, however, the central attribution of the syntactic effect to negative PMI rather than to the negative-sampling update stream is not supported by the experimental design.

major comments (2)

[§3–§4 and Table 1] The central sign-ablation comparison does not isolate the sign of PMI because the LexVec training stream is not a balanced sample of matrix cells. In §3, LexVec performs one window-sampled update per center word plus k=5 negative-sampling updates. In the +PPMI condition, every non-positive target is skipped, which removes all negative-sampling updates for non-cooccurring pairs (PPMI target 0); in the -CPMI-2 condition, those same updates are kept with target -2. The comparison therefore changes an optimization component—the synthetic negative-sampling stream, which is known to regularize representations—alongside the sign of the target. This is not a minor detail: Table 1 shows that +PPMI scores 50.3 on GSyn while the full PPMI model, which differs from +PPMI only by including those zero-target updates, scores 62.2. The authors should rerun the sign ablation with a matched update stream (for example, applying the same negative-sampling updates with target 0 in +PPMI and target -2 in -CPMI-2, or using window-sampled pairs only) before attributing the observed effects to the sign of PMI.
[§5 and Table 1] The claim that -PMI "contributes almost exclusively syntactic information" is difficult to reconcile with the paper's own syntactic benchmark GSyn, where -CPMI-2 obtains 7.1, far below +PPMI (50.3) and the full CPMI-2 model (65.1). The text in §5 supports the syntactic claim by citing POS, Dep, and TopC, where -PMI is near the full model, but it does not discuss GSyn; the footnote classifying GSyn as "morphological but ... syntactic in nature" does not resolve the discrepancy. If the conclusion is about sentence-level syntactic structure, it should be stated at that level; if it is about syntax in general, the GSyn result needs an explanation.

minor comments (4)

[§3] The choice of z=-2 is justified only by "we find ... consistently gives better results"; please document the values tried and show that the main qualitative conclusions are stable across z.
[§4–§5, Table 1] No variance or significance is reported; several comparisons used to argue for the adequacy of PPMI (e.g., PPMI vs CPMI-2 on SimLex, STSB, and POS) are within a point or less and could be noise.
[§5] The phrase "significantly worst" should be "significantly worse."
[Abstract and §5] The phrase "a low-rank, weighted factorization" suggests a class of methods, but all experiments use LexVec on one corpus; either test another factorization method or narrow the claim accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the PMI-sign claim is an empirical result benchmarked on external tasks, not derived from the definitions of +PMI or -PMI.

full rationale

The central claim ('most of the encoded semantics and syntax come from positive PMI, in contrast to -PMI which contributes almost exclusively syntactic information') is established by training LexVec on sign-specific PMI variants and probing the resulting embeddings on external semantic and syntactic benchmarks (SimLex, RW, GSem, STSB, WC, GSyn, POS, Dep, TopC). The conditions are explicit experimental manipulations: '+PPMI skips SGD steps when PPMI(w,c) <= 0' and '-CPMI-2 skips steps when CPMI-2(w,c) > 0'. Nothing in these definitions logically forces the observed performance pattern, so the finding is not self-definitional. The z=-2 threshold is tuned empirically and openly reported ('we find that z=-2 captures most of the negative spectrum and consistently gives better results'), but the qualitative dissociation between semantics and syntax does not rest on that specific value. The self-citations to LexVec (Salle et al., 2016) concern the factorization method, not the PMI-sign claim, and the results are validated against external benchmarks rather than reduced to the cited model's equations. The possible confound that +PPMI omits negative-sampling updates while -CPMI-2 keeps them is a methodological concern about what the comparison isolates, not a circularity in the derivation chain.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the modeling assumption that skipping SGD steps isolates PMI sign information, on the task taxonomy dividing semantics from syntax, and on the representativeness of LexVec. The CPMI threshold z is a free parameter. No new physical or conceptual entities are introduced.

free parameters (2)

CPMI clip threshold z = -2
Selected by hand after observing that z=-2 captures most of the negative spectrum and 'consistently gives better results' than smaller values; this is a data-dependent tuning choice.
LexVec default hyperparameters = window=2, dim=300, neg=5, smoothing=.75, lr=.025, neg_power=.75
Taken from the default LexVec configuration from prior work and used unchanged; no sensitivity analysis is presented in this paper.

assumptions (3)

ad hoc to paper The skip-step manipulation isolates the information contributed by each PMI sign region.
The paper assumes that training a LexVec model only on cells with PMI>0 (or PMI<=0) yields representations that reflect only positive (or negative) PMI information. This load-bearing assumption is introduced in Section 4.
domain assumption The evaluation tasks cleanly separate semantic from syntactic information.
Tasks such as GSyn are described as 'morphological but many categories test for POS relations and are therefore syntactic in nature'; this classification is assumed without validation. Section 4.
domain assumption LexVec's weighted factorization is representative of low-rank factorization of PMI matrices in general.
The paper justifies using LexVec over SVD but does not show the main result holds for other factorization methods. Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics." pith.science (2026). https://pith.science/paper/PFXKH3AC

@misc{pith2026190806941,
  author       = {Pith},
  title        = {Pith review of: Why So Down? The Role of Negative (and Positive) Pointwise Mutual Information in Distributional Semantics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PFXKH3AC}},
  note         = {Machine review of arXiv:1908.06941}
}

abstract

In distributional semantics, the pointwise mutual information ($\mathit{PMI}$) weighting of the cooccurrence matrix performs far better than raw counts. There is, however, an issue with unobserved pair cooccurrences as $\mathit{PMI}$ goes to negative infinity. This problem is aggravated by unreliable statistics from finite corpora which lead to a large number of such pairs. A common practice is to clip negative $\mathit{PMI}$ ($\mathit{\texttt{-} PMI}$) at $0$, also known as Positive $\mathit{PMI}$ ($\mathit{PPMI}$). In this paper, we investigate alternative ways of dealing with $\mathit{\texttt{-} PMI}$ and, more importantly, study the role that negative information plays in the performance of a low-rank, weighted factorization of different $\mathit{PMI}$ matrices. Using various semantic and syntactic tasks as probes into models which use either negative or positive $\mathit{PMI}$ (or both), we find that most of the encoded semantics and syntax come from positive $\mathit{PMI}$, in contrast to $\mathit{\texttt{-} PMI}$ which contributes almost exclusively syntactic information. Our findings deepen our understanding of distributional semantics, while also introducing novel $PMI$ variants and grounding the popular $PPMI$ measure.

Figures

Figures reproduced from arXiv: 1908.06941 by the authors.

**Figure 1.** CPMI -5 histogram (bucket width equal to .2) of 105 sampled pairs using window sampling and negative sampling. Number of samples in interval: [−5, −5] = 41695, (−5, 0] = 11001, [−2, 0] = 10759, (0, ∞) = 47304 Normalization: We also experiment with normalized PMI (NPMI) (Bouma, 2009): NPMI(w, c) = PMI(w, c)/ − log(Mwc/M∗∗) such that NPMI(w, c) = −1 when (w, c) never cooccur, NPMI(w, c) = 0 when they are independent… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 18 canonical work pages

[1]

Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, pages 31--40

work page 2009
[2]

John A Bullinaria and Joseph P Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior research methods, 39(3):510--526

work page 2007
[3]

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1--14

2017
[4]

Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22--29

work page 1990
[5]

Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070

arXiv 2018
[6]

Curran and Marc Moens

James R. Curran and Marc Moens. 2002. Improvements in automatic thesaurus extraction. In Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition - Volume 9, ULA '02, pages 59--66, Stroudsburg, PA, USA. Association for Computational Linguistics

work page 2002
[7]

John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis

work page 1957
[8]

Zellig S Harris. 1954. Distributional structure. Word

work page 1954

Show all 25 references

[9]

Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665--695

2015
[10]

Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. CoRR, abs/1508.01991

2015 arXiv
[11]

Dan Jurafsky. 2000. Speech & language processing. Pearson Education India

2000
[12]

Douwe Kiela and Stephen Clark. 2014. https://doi.org/10.3115/v1/W14-1503 A systematic study of semantic vector space model parameters . In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 21--30, Gothenburg, Sweden. Ass...

2014 doi
[13]

Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems, pages 2177--2185

2014
[14]

Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013, 104

2013
[15]

Christopher D Manning, Christopher D Manning, and Hinrich Sch \"u tze. 1999. Foundations of statistical natural language processing. MIT press

1999
[16]

Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. http://dl.acm.org/citation.cfm?id=972470.972475 Building a large annotated corpus of english: The penn treebank . Comput. Linguist., 19(2):313--330

1993
[17]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

2013 arXiv
[18]

Dmitrijs Milajevs, Mehrnoosh Sadrzadeh, and Matthew Purver. 2016. https://doi.org/10.18653/v1/P16-3009 Robust co-occurrence quantification for lexical distributional semantics . In Proceedings of the ACL 2016 Student Research Workshop, pages 58--64, Berlin, Germany. Associatio...

2016 doi
[19]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), 12

2014
[20]

Tamara Polajnar and Stephen Clark. 2014. https://doi.org/10.3115/v1/E14-1025 Improving distributional semantic vectors through context selection and normalisation . In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, ...

2014 doi
[21]

Terry Regier and Susanne Gahl. 2004. Learning the unlearnable: The role of missing evidence. Cognition, 93(2):147--155

2004
[22]

Alexandre Salle, Aline Villavicencio, and Marco Idiart. 2016. http://aclweb.org/anthology/P/P16/P16-2068.pdf Matrix factorization using window sampling and negative sampling for improved word representations . In Proceedings of the 54th Annual Meeting of the Association for Co...

2016
[23]

Noam Shazeer, Ryan Doherty, Colin Evans, and Chris Waterson. 2016. Swivel: Improving embeddings by noticing what's missing. arXiv preprint arXiv:1602.02215

2016 arXiv
[24]

Peter D Turney and Michael L Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems (TOIS), 21(4):315--346

2003
[25]

Xin Xin, Yuan Fajie, and He Xiangnan. 2018. Batch is not heavy: Learning word embeddings from all samples. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL, pages 31--40

work page 2009

[2] [2]

John A Bullinaria and Joseph P Levy. 2007. Extracting semantic representations from word co-occurrence statistics: A computational study. Behavior research methods, 39(3):510--526

work page 2007

[3] [3]

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1--14

2017

[4] [4]

Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics, 16(1):22--29

work page 1990

[5] [5]

Alexis Conneau, German Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. arXiv preprint arXiv:1805.01070

arXiv 2018

[6] [6]

Curran and Marc Moens

James R. Curran and Marc Moens. 2002. Improvements in automatic thesaurus extraction. In Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition - Volume 9, ULA '02, pages 59--66, Stroudsburg, PA, USA. Association for Computational Linguistics

work page 2002

[7] [7]

John R Firth. 1957. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis

work page 1957

[8] [8]

Zellig S Harris. 1954. Distributional structure. Word

work page 1954