{"id":"a82b3be4-cb0f-48f0-88d7-cfa696bf853f","arxiv_id":"1908.06941","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"In word embeddings built by factorizing PMI matrices, positive PMI carries most semantic and syntactic information, while negative PMI contributes almost solely syntactic information.","lead":"This paper studies how the positive and negative parts of a common statistical word co-occurrence score (PMI) each contribute to word embeddings, and finds that the positive part carries most meaning while the negative part mostly signals grammar. It matters because it explains why the widely used PPMI shortcut works, and suggests when keeping negative information could help.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The +PPMI vs -CPMI-2 comparison conflates PMI sign with LexVec's negative-sampling procedure, so the syntactic dissociation may be an optimization artifact rather than evidence about matrix content.","rationale":"The paper is a well-designed empirical study, and the conditional verdict is appropriate. My stress-test focused on whether the sign-filtering method isolates the information content of PMI signs. It does not do so as cleanly as claimed, because LexVec's training stream is a mixture of window-sampled and negative-sampled pairs, and the +PPMI condition filters out all negative-sampled updates while the -CPMI-2 condition keeps them. The Table 1 contrast between +PPMI and full PPMI (GSem 56.1 vs 80.3; GSyn 50.3 vs 62.2) shows that adding zero-target negative-sampling updates changes exactly the tasks where the paper hypothesizes -PMI matters. This is a concrete, load-bearing confound: the conclusion that negative PMI contributes almost exclusively syntactic information may be a statement about negative sampling rather than about the negative spectrum of PMI. I do not think this invalidates the paper, because the effect is real under the LexVec training regime and the authors are careful to frame their results empirically, but the central claim needs the proposed balanced-factorization control. The reader's weakest_assumption pointed to update counts; I agree only partially, since the paper's own histogram suggests the sign ratio in the training stream is not strongly unbalanced. The sharper issue is the qualitative difference between observed positive cooccurrence updates and synthetic negative-sampling updates. Because this concern is addressable with a clean control and the paper already merits a conditional verdict, I leave the verdict unchanged.","tokens_in":6502,"tokens_out":10657,"duration_ms":112096,"concrete_test":"Perform the sign-filtering comparison with a sampling scheme that equalizes optimization exposure: from the same cooccurrence counts, construct sign-balanced PMI matrices containing equal numbers of positive and negative entries, and factorize them with a fixed weighted least-squares objective (e.g., truncated SVD) using the same embedding dimension. Evaluate +PMI-only, -PMI-only, and full matrices on SimLex, RW, GSem, GSyn, POS, Dep, and TopC. If the -PMI-only model still gives near-random semantic scores but POS and probing scores comparable to the full model, the dissociation is intrinsic to the matrix content; if the syntactic advantage disappears, the paper's central claim is an artifact of LexVec's negative sampling and update counts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that +PMI carries semantics and syntax while -PMI is almost exclusively syntactic. To test this, Section 4 trains LexVec models that skip SGD steps when the target is positive or non-positive. But LexVec's training stream (Section 3) contains two kinds of updates: window-sampled observed pairs and k negative-sampled non-cooccurring pairs per center word. In the +PPMI condition, every non-positive target is skipped, including all negative-sampling updates (PPMI=0). In the -CPMI-2 condition, non-positive targets are kept, so the model receives the full negative-sampling stream with clipped target -2. The comparison therefore varies the optimization procedure together with the information source: +PPMI has no synthetic negative examples, while -CPMI-2 is trained with a large number of them. Negative sampling is an optimization component, not merely a vehicle for negative PMI, and it is known to act as a regularizer and to improve analogy and rare-word performance. Table 1 shows this directly: +PPMI gets 56.1 and 50.3 on GSem and GSyn, while full PPMI, which only adds zero-target negative-sampling updates, jumps to 80.3 and 62.2. Thus the syntactic contribution attributed to -PMI may come from negative sampling with a clipped target, not from the sign of PMI as an information channel. The reader's update-count concern is related, but Figure 1 suggests the sampled stream is not strongly unbalanced across signs; the sharper problem is the asymmetric role of negative sampling in the two conditions.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies the roles of positive and negative pointwise mutual information in low-rank factorization of PMI-weighted co-occurrence matrices. It proposes two PMI variants, CPMI_z (clipping PMI from below at a value z, recovering PPMI for z=0) and NNEGPMI (normalizing only the negative part of PMI), and evaluates them with the LexVec factorization model. To separate the contributions of the two signs, the authors train LexVec with SGD updates skipped when the target cell value is non-positive (+PPMI condition) or positive (-CPMI-2/-NNEGPMI conditions). On semantic and syntactic benchmarks, +PPMI performs well on both semantic and syntactic tasks, whereas the negative-only models perform near baseline on semantics but close to full models on POS tagging and the SentEval syntactic probing tasks. The paper concludes that most semantics and syntax come from +PMI, that -PMI carries almost exclusively syntactic information, and that PPMI is a well-grounded choice because full-spectrum models perform no better.","tokens_in":6828,"tokens_out":13281,"duration_ms":138141,"significance":"The question is relevant for embedding design and for the theory of distributional semantics. If the main empirical claim were established, the finding would be practically useful: it would justify discarding negative PMI for semantic tasks and might allow negative PMI to be treated as a syntactic signal. The paper is a clearly written empirical study with a broad task battery (word similarity, analogies, sentence similarity, probing tasks, POS tagging), introduces two transparent PMI variants, and releases code. The main empirical claim is falsifiable, and the full-versus-ablated comparisons in Table 1 are easy to reproduce. At present, however, the central attribution of the syntactic effect to negative PMI rather than to the negative-sampling update stream is not supported by the experimental design.","major_comments":[{"comment":"The central sign-ablation comparison does not isolate the sign of PMI because the LexVec training stream is not a balanced sample of matrix cells. In §3, LexVec performs one window-sampled update per center word plus k=5 negative-sampling updates. In the +PPMI condition, every non-positive target is skipped, which removes all negative-sampling updates for non-cooccurring pairs (PPMI target 0); in the -CPMI-2 condition, those same updates are kept with target -2. The comparison therefore changes an optimization component—the synthetic negative-sampling stream, which is known to regularize representations—alongside the sign of the target. This is not a minor detail: Table 1 shows that +PPMI scores 50.3 on GSyn while the full PPMI model, which differs from +PPMI only by including those zero-target updates, scores 62.2. The authors should rerun the sign ablation with a matched update stream (for example, applying the same negative-sampling updates with target 0 in +PPMI and target -2 in -CPMI-2, or using window-sampled pairs only) before attributing the observed effects to the sign of PMI.","section":"§3–§4 and Table 1"},{"comment":"The claim that -PMI \"contributes almost exclusively syntactic information\" is difficult to reconcile with the paper's own syntactic benchmark GSyn, where -CPMI-2 obtains 7.1, far below +PPMI (50.3) and the full CPMI-2 model (65.1). The text in §5 supports the syntactic claim by citing POS, Dep, and TopC, where -PMI is near the full model, but it does not discuss GSyn; the footnote classifying GSyn as \"morphological but ... syntactic in nature\" does not resolve the discrepancy. If the conclusion is about sentence-level syntactic structure, it should be stated at that level; if it is about syntax in general, the GSyn result needs an explanation.","section":"§5 and Table 1"}],"minor_comments":[{"comment":"The choice of z=-2 is justified only by \"we find ... consistently gives better results\"; please document the values tried and show that the main qualitative conclusions are stable across z.","section":"§3"},{"comment":"No variance or significance is reported; several comparisons used to argue for the adequacy of PPMI (e.g., PPMI vs CPMI-2 on SimLex, STSB, and POS) are within a point or less and could be noise.","section":"§4–§5, Table 1"},{"comment":"The phrase \"significantly worst\" should be \"significantly worse.\"","section":"§5"},{"comment":"The phrase \"a low-rank, weighted factorization\" suggests a class of methods, but all experiments use LexVec on one corpus; either test another factorization method or narrow the claim accordingly.","section":"Abstract and §5"}],"recommendation":"major_revision","confidential_remarks":"The negative-sampling confound is the key issue for the paper's central claim. The released code and the broad task battery are strengths, and the paper could be suitable for reconsideration after a revision that adds matched-stream ablations and error bars. The GSyn discrepancy should also be addressed before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper asks a good question — what negative PMI actually contributes in low-rank factorization of PMI matrices — and it runs a wide battery of probes (SimLex, RW, analogies, STSB, SentEval, POS) to answer it. The headline finding, that +PMI alone captures most semantics and syntax while -PMI-only models look almost purely syntactic, is plausible and would be useful if it held. The paper also introduces two clean variants, CPMI_z and NNEGPMI, and ships code, which is credit-worthy.\n\nBut the central comparison is confounded, and the stress-test note is right about the sharper version of the problem. LexVec's training stream is not just the cooccurrence-weighted cells; it contains k negative-sampling updates per center word. In the +PPMI condition, every non-positive target is skipped, so all negative-sampling updates (target 0) are dropped. In the -CPMI-2 condition, non-positive targets are kept, so the model receives the full negative-sampling stream with clipped target -2. The two conditions therefore differ in the presence of negative sampling as an optimization component, not merely in the sign of the PMI signal. The paper's own Table 1 shows the size of this effect: +PPMI scores 56.1 on GSem and 50.3 on GSyn, while full PPMI — which only adds the zero-target negative-sampling updates — jumps to 80.3 and 62.2. Attributing that jump to \"negative PMI information\" is not supported, since the target is 0 in those updates. Negative sampling is a known regularizer and rare-word improver; the \"syntactic\" behavior of -CPMI-2 may well be that optimizer artifact, not evidence about matrix content.\n\nThe reader's update-count concern is related, but Figure 1 suggests the sampled stream is not wildly unbalanced by sign; the asymmetric role of negative sampling is the more decisive problem.\n\nWhat survives is narrower: within LexVec with its default optimization, clipping negative PMI at 0 (PPMI) is just as good as more elaborate treatments, and full NPMI hurts. The paper does not establish the semantic/syntactic dissociation. A clean control would be to factor the matrix with SVD (no negative sampling) or to add negative-sampling updates with target 0 to the +PPMI condition. Absent that, I'd treat the main claim as unproven.\n\nWho is this for? Researchers working on embedding objectives and association measures. It deserves a serious referee, but the referee should ask for the control. I'd send it to review, with the expectation of heavy revision.\n\nRecommendation: engage with it, but only after the confound is addressed. The question is worth getting right.","headline":"A useful probe of PMI weighting variants under LexVec, but the central semantic/syntactic dissociation is confounded by the asymmetric use of negative sampling.","tokens_in":7334,"tokens_out":2692,"would_cite":false,"duration_ms":26685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In low-rank factorization of PMI matrices, positive PMI carries nearly all semantic information, while negative PMI contributes almost exclusively syntactic information.","keywords":["pointwise mutual information","positive PMI","negative PMI","PPMI","word embeddings","matrix factorization","distributional semantics","syntactic information"],"falsifier":"Train a negative-PMI-only model with enough extra epochs or negative-cell sampling that it receives the same number of gradient updates as the positive-PMI-only model, and check whether its semantic-task scores remain near chance; if they rise substantially, the original sign-asymmetry conclusion is confounded by update count.","tokens_in":84,"feed_emoji":"💬","tokens_out":8538,"duration_ms":141618,"temperature":0.7,"pith_summary":"Pointwise mutual information (PMI) is the standard re-weighting for word co-occurrence counts, but its negative values are usually clipped to zero because they include many unreliable counts from finite corpora. This paper asks what that clipping actually loses. Using a low-rank factorization of PMI matrices and probing the resulting embeddings on semantic and syntactic tasks, it argues that positive PMI carries nearly all semantic information and most syntactic information, while negative PMI carries almost exclusively syntactic information. If the claim holds, the common PPMI shortcut is not just a numerical convenience; it separates meaning from grammar, and negative PMI becomes an available unsupervised signal for syntax.","feed_headline":"Negative word associations carry syntax, not meaning","feed_subtitle":"Clipping the negative half of word-association scores costs almost no meaning, leaving a pure syntactic signal.","key_machinery":"The machinery is the family of clipped PMI matrices $CPMI_z(w,c) = \\max(z, PMI(w,c))$, where $z=0$ gives PPMI and a small negative threshold such as $z=-2$ captures most of the negative spectrum, together with a sign-restricted stochastic gradient training scheme that skips updates for target cells whose value has the wrong sign. That scheme produces separate positive-only and negative-only models from the same factorization framework, so differences on downstream probes can be attributed to the information carried by each sign. A supplementary variant, NNEGPMI, normalizes only the negative branch of PMI, testing whether scale in the negative spectrum matters.","core_discovery":"The paper's central claim is that the sign of PMI splits distributional information by type. In a low-rank factorization trained with window sampling and negative sampling, a model that only sees non-negative PMI values performs as well as or better than full-spectrum models on nearly every semantic task and on most syntactic probes; a model that only sees negative PMI values performs near chance on semantic tasks but approaches full-model accuracy on part-of-speech tagging and syntactic sentence probes from the same architecture. The authors conclude that positive PMI encodes both semantics and syntax, negative PMI encodes almost exclusively syntax, and that clipping the negative spectrum to zero, as PPMI does, discards little of what matters for meaning.","pith_inferences":["A direct testable extension is a rare-word hybrid: add clipped negative PMI only for low-frequency words, predicting it beats both pure PPMI and full-spectrum models by giving rare words repulsion without contaminating common-word semantics.","If negative PMI really encodes syntax, then PPMI-only embeddings should show characteristic blind spots on syntactic probing tasks, such as near-chance scores on agreement or word-order probes, while mixed models do not.","The paper's connection to indirect negative evidence suggests a cross-linguistic prediction: languages with richer inflectional morphology should show stronger syntactic signal in negative PMI than analytic languages, because agreement constraints create more systematic non-cooccurrence patterns."],"forward_implications":["PPMI embeddings are sufficient when the goal is semantic similarity or textual meaning, since the clipped negative spectrum contributes little to those tasks.","Negative PMI can be treated as an unsupervised syntax channel, useful for tasks that need word-order, agreement, or part-of-speech information without labeled syntactic data.","Scale information in the negative spectrum is not needed: clipping negatives at $-2$ or normalizing only the negative branch preserves most performance, while normalizing the full PMI spectrum hurts.","The advantage of full-spectrum models on rare-word similarity and analogies suggests negative PMI should be exploited selectively for rare words rather than applied globally.","Because positive PMI alone matches full models on most tasks, embedding pipelines can skip negative-PMI computation and reduce memory and update cost without losing semantic content."],"supporting_citations":[{"why":"Introduces normalized PMI (NPMI), the baseline variant that the paper's NNEGPMI adapts and compares against.","marker":"(Bouma, 2009)"},{"why":"Shows skip-gram embeddings implicitly factorize a shifted PMI matrix, motivating explicit low-rank PMI factorization as the test bed.","marker":"(Levy and Goldberg, 2014)"},{"why":"Provides the window-sampling, negative-sampling factorization algorithm and configuration used to train every model.","marker":"(Salle et al., 2016)"},{"why":"Supplies negative sampling, subsampling, and the Google semantic and syntactic analogy datasets used as probes.","marker":"(Mikolov et al., 2013)"},{"why":"Supplies SimLex, the word-similarity probe that distinguishes semantic from associative similarity.","marker":"(Hill et al., 2015)"},{"why":"Supplies the Rare Word dataset used to test rare-word semantic representations.","marker":"(Luong et al., 2013)"},{"why":"Supplies the WC, Depth, and Top Constituent probing tasks used for sentence-level semantics and syntax.","marker":"(Conneau et al., 2018)"},{"why":"Supplies the Penn Treebank WSJ section used for the part-of-speech tagging probe.","marker":"(Marcus et al., 1993)"}],"fun_headline_variants":["Negative word scores are syntax, not meaning","PMI's negative half: all syntax, little semantics","Clipping negative PMI loses syntax, not meaning","Positive PMI drives semantics; negative PMI drives syntax","Word-association sign splits syntax from meaning"],"cache_read_input_tokens":9472,"weakest_assumption_plain":"The experiments assume that skipping stochastic gradient updates for cells whose PMI has the wrong sign isolates the information carried by the other sign, but because positive cells appear far more often in the training stream than negative cells, the sign-specific models receive very different numbers of updates, so the observed differences may partly reflect training quantity rather than information content.","fun_headline_variants_meta":{"raw":{"variants":["Negative word scores are syntax, not meaning","PMI's negative half: all syntax, little semantics","Clipping negative PMI loses syntax, not meaning","Positive PMI drives semantics; negative PMI drives syntax","Word-association sign splits syntax from meaning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2276,"prompt_tokens":929,"completion_tokens":1347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":1273}},"tokens_in":545,"tokens_out":1347,"duration_ms":9037,"temperature":1.0,"reasoning_tokens":1273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:29:56.346789+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a negative-PMI-only model with enough extra epochs or negative-cell sampling that it receives the same number of gradient updates as the positive-PMI-only model, and check whether its semantic-task scores remain near chance; if they rise substantially, the original sign-asymmetry conclusion is confounded by update count.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces normalized PMI (NPMI), the baseline variant that the paper's NNEGPMI adapts and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows skip-gram embeddings implicitly factorize a shifted PMI matrix, motivating explicit low-rank PMI factorization as the test bed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the window-sampling, negative-sampling factorization algorithm and configuration used to train every model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SimLex, the word-similarity probe that distinguishes semantic from associative similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Rare Word dataset used to test rare-word semantic representations."}],"review_version":1}