{"id":"6dfba314-4f6d-466b-8b7b-d96dacda8e0b","arxiv_id":"1909.01772","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Aff2Vec affect-enriched embeddings yield small ranking improvements on TREC Core 2017 news data, but gains do not generalize to query expansion or to the CACM collection.","lead":"This paper evaluates Aff2Vec, word embeddings enriched with emotion scores, for news search ranking and query expansion. It finds modest gains on New York Times data but mixed results elsewhere, so the practical value is real but narrow.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The alleged affect benefit is confounded with base-embedding identity: affect-enriched rows are not matched to their vanilla counterparts in corpus, dimensionality, vocabulary, or preprocessing, so the NDCG gains could come from the base vectors rather than affect.","rationale":"The reader's weakest_assumption is the same one that I find most load-bearing: the comparison does not isolate affect enrichment. I agree with the CONDITIONAL verdict because the paper does show some plausible positive signals, especially for ranking on NYT, but the absence of matched base embeddings, significance testing, and a stated expansion threshold (Section 3) prevents the central claim from being accepted as established. I did not find a reason to move to REJECT: the tables contain several affect-versus-vanilla pairs that may be matched in the original Aff2Vec resources, and the controlled test I propose could validate the effect. I also did not find a reason to move to ACCEPT, since no such validation appears in the paper. I additionally note that the query expansion threshold t in Section 3 is never given a numeric value, so all query expansion results are conditional on an unreported hyperparameter; this is a reproducibility issue secondary to the base-embedding confound. The verifiable fix is straightforward and would settle the attribution question, so the appropriate target remains the reader's conditional assessment.","tokens_in":8213,"tokens_out":8047,"duration_ms":82791,"concrete_test":"One controlled experiment: build the exact base checkpoint for GloVe-affect (state source file, dimension, corpus, and preprocessing), then run the Section 4 AWE ranking pipeline with that base and with GloVe-affect under identical tokenization, stopwords, TF-IDF normalization, and OOV handling, and compute a paired bootstrap or randomization test over the 50 TREC-2017 topics. Require a positive, significant NDCG delta before attributing the gain to affect. If the original base checkpoint is not identified, reconstruct the control by taking one GloVe checkpoint and applying the Aff2Vec retrofitting procedure with and without the affect lexicon, holding iterations and all other settings fixed, and compare those two models on the same pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim requires that affect enrichment, and not the underlying embedding, causes the reported IR gains. That condition is not established anywhere. In Section 5.1, Table 2, the best ranking model GloVe-retrofitted-affect-555 (NDCG 0.4693) is compared against GloVe-retrofitted (0.4216) and GloVe (0.4292), but the authors never state that the affect variant and its base share the same corpus, dimensionality, vocabulary, tokenization, or preprocessing. The same problem appears for GloVe-affect versus GloVe and paragram-affect versus paragram in Tables 2-5. Section 2.1 only cites [KCC18]; it does not specify which exact vanilla checkpoint each affect model was built on. Section 4 then averages word embeddings with TF-IDF normalization, so any difference in embedding vocabulary or OOV handling changes which terms contribute to the query and document vectors; if the affect model covers more NYT vocabulary, the AWE representation is not comparable even with the same embedding dimension. The conclusion's '~20%' improvement (Section 6) is not supported by the tables (best NDCG gain over BM25 is about 8%, over vanilla GloVe about 9%), which is consistent with the effect being at least partly an artifact of unmatched baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to apply Aff2Vec affect-enriched word embeddings to two information retrieval tasks, ranking and query expansion, on the TREC Core 2017 New York Times dataset and the CACM classic collection. It compares several Aff2Vec variants (affect, counterfitted, retrofitted) against vanilla embeddings, Dict2vec, WordNet, and BM25/QL baselines using Anserini and Lucene4IR, reporting NDCG and MAP. The main claimed result is that affect-enriched embeddings, notably GloVe-retrofitted-affect-555, improve ranking NDCG on NYT over BM25 and vanilla embeddings, while results on CACM are mixed and query expansion on CACM is best served by a vanilla word2vec model.","tokens_in":8476,"tokens_out":5391,"duration_ms":47769,"significance":"If the causal effect claimed by the authors held, this would be a practical contribution: injecting affect information into embeddings would improve news IR at low implementation cost. The paper benefits from using externally judged TREC relevance assessments, multiple embedding baselines, and standard open-source IR toolkits, and it reports both NDCG and MAP. However, the evaluation does not currently support the causal interpretation because affect-enriched and vanilla models are not matched on base corpus, dimensionality, or vocabulary; there are no significance tests; and the query-expansion threshold is unreported. These issues materially weaken the conclusion.","major_comments":[{"comment":"The central comparison is confounded: affect-enriched models are not matched to their vanilla counterparts in base corpus, dimensionality, vocabulary, or preprocessing. For example, GloVe-retrofitted-affect-555 (NDCG 0.4693) is compared with GloVe-retrofitted (0.4216) and GloVe (0.4292), but the text never states which GloVe checkpoint the affect variant was built from; Table 2's GloVe variants span 6B, 42B, and Twitter corpora and dimensions 25–300. Because Section 4 averages embeddings with TF-IDF normalization, different vocabulary coverage changes which terms are included in the query and document vectors, so the observed differences cannot be attributed specifically to affect enrichment. The authors should either explicitly match the base embeddings or provide a matched comparison, and report vocabulary overlap and OOV handling.","section":"§2.1, §5.1, Tables 2–5"},{"comment":"The cosine threshold t for query expansion is never reported in the experiments, despite being a free parameter with direct impact on the number and quality of expansion terms. Without t, the query expansion results are not reproducible, and it is unknown whether the reported gains are sensitive to this parameter. Moreover, several rows in Table 3 report identical NDCG/MAP values (e.g., 0.4308/0.1947 for GloVe.42B.300d and GloVe-affect), which suggests that few or no expansions were applied; the authors should report how many queries were expanded and explain these duplicate values.","section":"§3, Tables 3 and 5"},{"comment":"The conclusion states a 'significant improvement' of roughly 20% over vanilla embeddings, but the tables show at most an 8–9% NDCG gain over BM25 or GloVe on NYT ranking (0.4693 vs 0.4334 and 0.4292), and on CACM the affect gains are not consistent. The paper reports no significance tests, confidence intervals, or error bars for any NDCG/MAP difference; given the small absolute differences, the observed improvements may be within noise. The authors should run paired statistical tests over the query/topic sets, report the exact p-values, and soften the overclaim.","section":"§6"},{"comment":"The preliminary motivation for focusing on news uses the Aff2Vec affect scoring algorithm itself to compute formality, politeness, and frustration scores on the collections, and then asserts that the resulting scores 'suggest that Aff2Vec embeddings should work well' on news. This is self-referential: the same tool being evaluated is used as evidence for its own suitability. The paper should either use an independent affect measure or external evidence, or explicitly frame Table 1 as an illustration with no evidential weight.","section":"§1.1"}],"minor_comments":[{"comment":"'we except Aff2Vec models to work well' should read 'we expect Aff2Vec models to work well'.","section":"Introduction, Section 1"},{"comment":"'Ney Work Times' is a typo for 'New York Times'.","section":"Section 1.1"},{"comment":"'paragaram' is a typo for 'paragram' in the sentence introducing the CACM results.","section":"Section 5.1"},{"comment":"[TGH17a] and [TGH17b] are the same Dict2vec paper; this duplicate should be consolidated.","section":"References"},{"comment":"The statement 'we used each word TF-iDF vector to normalize (divide) the averaged word embedding' is unclear. If a TF-IDF weight is used to scale each term's embedding before averaging, please state the exact formula; if the TF-IDF vector is used as a denominator, please define the division operation.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"For the editor: This is a modest empirical study whose main claim is currently under-supported. The confounded comparisons and missing significance tests are the key blockers; a focused revision that adds matched baselines and statistical tests could make the paper acceptable for a journal. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable first test of whether affect-enriched word embeddings help news IR, with genuinely mixed results. The best NYT ranking gain (GloVe-retrofitted-affect-555, NDCG 0.4693 vs BM25 0.4334) is about 8% relative, which is useful if it is real. But the paper never isolates affect enrichment from the identity of the base embedding, so the main claim is not yet established.\n\nWhat is new: it is the first evaluation of Aff2Vec embeddings for query expansion and ranking, on two standard collections (NYT TREC Core '17 and CACM), using standard toolkits. The comparison is broad—many pretrained vanilla, enriched, and affect-enriched models—and the tables are honestly mixed: on NYT ranking, affect models beat BM25 on NDCG but not MAP; on CACM query expansion, vanilla word2vec wins. Credit where due: the authors do not oversell the CACM query expansion results.\n\nSoft spots, in proportion. The biggest one is the confounded comparison. Affect-enriched rows (GloVe-affect, paragram-affect, etc.) are not matched to their vanilla counterparts in base corpus, dimensionality, vocabulary, or preprocessing. The paper never specifies which exact vanilla checkpoint each affect model was built on, so the gains could come from the base vectors rather than affect information. That is a load-bearing flaw, especially since some non-affect enriched models also do very well on CACM. Second, there are no significance tests or error bars anywhere; differences of 0.01 to 0.04 NDCG could easily be noise. Third, the query expansion threshold t is never reported, so the expansion results are not reproducible. Fourth, the conclusion's claim of ~20% improvement over vanilla embeddings is not supported by the tables—the best NDCG gain over GloVe is about 9%, over BM25 about 8%. Fifth, the motivation in Section 1.1 uses the Aff2Vec affect-scoring algorithm itself to argue that news text is high in affect, which is circular; it tells you little about whether affect enrichment helps retrieval.\n\nOverall, the paper is a useful experimental starting point, but the central claim is conditionally supported at best. A corrected version with matched base embeddings, significance tests, reported t, and an honest summary would make it defensible. I'd send it to peer review—it deserves referee time—but I'd expect substantial revision. I would not cite it as evidence for affect enrichment until the confound is fixed.","headline":"Useful first test of affect-enriched embeddings for news IR, but the central claim is confounded by unmatched base embeddings; worth refereeing with major revision.","tokens_in":8996,"tokens_out":3562,"would_cite":false,"duration_ms":34364,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Affect-enriched word embeddings, built by injecting valence, arousal, and dominance scores into existing vector spaces, improve news ranking and query expansion over vanilla embeddings and BM25 on New York Times data.","keywords":["affect-enriched word embeddings","Aff2Vec","news information retrieval","query expansion","document ranking","valence arousal dominance","synonym-antonym distinction","TREC Core 2017"],"falsifier":"Repeat the NYT ranking and query expansion evaluations with affect-enriched and vanilla embeddings trained on the same corpus at the same dimensionality, or shuffle the affect scores before enrichment; if GloVe-retrofitted-affect-555 no longer beats GloVe and BM25, the causal role of affect information is not established.","tokens_in":7990,"feed_emoji":"📰","tokens_out":6380,"duration_ms":58096,"temperature":0.7,"pith_summary":"This paper claims that injecting affect information into word embeddings improves information retrieval for news text. It uses Aff2Vec, a post-training method that adds valence, arousal, and dominance scores from Warriner's lexicon into existing embeddings such as GloVe and word2vec. In ranking experiments on New York Times articles, the best affect-enriched model, GloVe-retrofitted-affect-555, reaches NDCG 0.4693 versus 0.4334 for BM25; similar gains appear in query expansion. On the CACM collection, affect-enriched paragram embeddings give the best ranking results, but plain word2vec wins query expansion, suggesting the benefit depends on the domain.","feed_headline":"Affect-Aware Vectors Beat BM25 on News Ranking","feed_subtitle":"Injecting valence, arousal, and dominance into word embeddings lifts retrieval quality on New York Times news.","key_machinery":"The central mechanism is Aff2Vec's post-training affect injection: starting from GloVe, word2vec, or paragram vectors, it adjusts embeddings with valence, arousal, and dominance scores from the Warriner lexicon; retrofitting and counterfitting are used to create the variants. In the retrieval pipeline, query and document vectors are TF-IDF-normalized averages of these word vectors, and ranking uses cosine similarity; query expansion replaces query terms with nearest neighbours above a cosine threshold, producing OR queries.","core_discovery":"The central discovery is that affect-enriched term embeddings transfer gains to standard IR tasks, at least in the news domain. The authors evaluate ranking and query expansion using pretrained vanilla and enriched embeddings. They find that affect-enriched variants outperform their unenriched counterparts in most news settings: the top model, GloVe retrofitted with affect information (GloVe-retrofitted-affect-555), obtains NDCG 0.4693 in ranking and 0.4421 in query expansion on NYT, beating BM25's 0.4334. The paper interprets this as evidence that capturing affective attitudes in news text, and separating antonyms such as 'happy' and 'sad', makes embeddings better aligned with news relevance.","pith_inferences":["Because the paper reports no statistical significance tests, a paired bootstrap over per-topic NDCG scores would show whether the roughly 0.03 NDCG gap over BM25 is real or within noise; this is an editorial caution, not a paper claim.","A testable extension: measure a dataset's formality, politeness, and frustration scores as the paper does for NYT, WP, CACM, and ClueWeb09, and check whether the size of the affect-enrichment gain predicts those scores; the paper motivates, but does not test, this link.","The paper's own numbers suggest BM25 still wins on MAP for NYT ranking; a fusion of BM25 scores with affect-embedding cosine scores is a natural next step, though the paper does not propose it.","If affect enrichment works by separating antonym pairs, gains should concentrate on queries and documents with affective vocabulary; examining per-query differences between affect and vanilla embeddings would expose the mechanism."],"forward_implications":["Affect-enriched term embeddings can be used in place of vanilla embeddings in standard retrieval pipelines, giving higher NDCG on news ranking without changing the underlying retrieval model.","Nearest-neighbour query expansion built from affect-enriched embeddings outperforms both classic BM25 and WordNet-based expansion on the New York Times dataset.","The top performing model, GloVe-retrofitted-affect-555, shows the benefit survives when affect information is injected into a strong base embedding rather than learned from scratch.","The benefit is domain-dependent: on the CACM collection, affect-enriched paragram embeddings lead ranking, but vanilla word2vec leads query expansion, so news-like affective content is where the approach pays off."],"supporting_citations":[{"why":"Defines Aff2Vec, the affect-enriched embedding approach central to the paper.","marker":"[KCC18]"},{"why":"Provides GloVe embeddings that serve as the base for the top-performing affect-enriched models.","marker":"[PSM14]"},{"why":"Provides word2vec embeddings used as both baseline and base for affect enrichment.","marker":"[MCCD13]"},{"why":"Supplies the valence, arousal and dominance lexicon that carry affect information into the embeddings.","marker":"[WKB13]"},{"why":"Retrofitting procedure used to build the best-ranking GloVe-retrofitted-affect model.","marker":"[FDJ+15]"},{"why":"Counter-fitting procedure used to create counterfit variants of the affect embeddings.","marker":"[MOT+16]"},{"why":"Query expansion approach with nearest-neighbour terms that the paper adapts for its expansion experiments.","marker":"[RPMG16]"},{"why":"Defines the New York Times TREC Core 2017 benchmark that supplies the main news test collection.","marker":"[AHK+17]"}],"fun_headline_variants":["Affect-Enriched Embeddings Outperform BM25 on News","Injecting Emotion into Word Vectors Boosts News Search","Aff2Vec: Emotion-Tuned Embeddings Win News Retrieval","Happy vs Sad: Affect-Aware Vectors Improve News Ranking","Emotion Awareness Lifts News Retrieval Beyond BM25"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes its ranking and expansion gains to affect enrichment, but the affect-enriched and vanilla embeddings it compares are not matched for training corpus, dimensionality, or vocabulary; the improvements could come from those base differences instead.","fun_headline_variants_meta":{"raw":{"variants":["Affect-Enriched Embeddings Outperform BM25 on News","Injecting Emotion into Word Vectors Boosts News Search","Aff2Vec: Emotion-Tuned Embeddings Win News Retrieval","Happy vs Sad: Affect-Aware Vectors Improve News Ranking","Emotion Awareness Lifts News Retrieval Beyond BM25"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2719,"prompt_tokens":836,"completion_tokens":1883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":1797}},"tokens_in":452,"tokens_out":1883,"duration_ms":13246,"temperature":1.0,"reasoning_tokens":1797,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:07:27.317986+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the NYT ranking and query expansion evaluations with affect-enriched and vanilla embeddings trained on the same corpus at the same dimensionality, or shuffle the affect scores before enrichment; if GloVe-retrofitted-affect-555 no longer beats GloVe and BM25, the causal role of affect information is not established.","supporting_citations":[],"review_version":1}