{"id":"dd8c0b93-7f4d-4e22-be73-6990efa756ad","arxiv_id":"2501.03475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using an LLM fine-tuned to translate text between 11 emotions, the authors neutralize sarcastic passages for RAG question answering, improving accuracy on fully sarcastic context by about 3 percentage points.","lead":"This paper builds a dataset of Wikipedia passages rewritten in 11 emotional tones, trains a model to translate between emotions, and tests whether converting sarcastic passages to a neutral tone helps LLMs answer questions. It reports a ~3% improvement on fully sarcastic passages, but the effect does not hold on passages with distorted facts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2.8% NQ-FS improvement rests on an untested assumption: one-way neutralization preserves facts. Section 6.2 validates round-trip human text, not the neutralized benchmark passages; QA accuracy with 10 passages cannot rule out factual corruption. A direct fact-preservation audit would settle it.","rationale":"The paper has real strengths: a large multi-emotion synthetic dataset, a multi-LLM generation procedure that reduces single-model bias, a fine-tuned translator that human raters prefer for emotional expressiveness, and a downstream experiment across four LLMs. The central application claim, however, is only as good as the fact-preservation guarantee. Section 6.2's round-trip evaluation on human-written text is an out-of-distribution proxy: it tests human text, uses a round trip, and measures reconstruction relative to an unfine-tuned LLM, not the one-way sarcasm-to-neutral transformation on the NQ-FS passages used in Section 6.1. The reader's weakest_assumption correctly identified this gap. I do not see an internal inconsistency in the main construction, and the concern is empirical rather than formal. The recommended conditional verdict stands: the paper should be accepted only if the fact-preservation check passes or the claims are narrowed to 'QA improves after neutralization regardless of fact fidelity.' If the audit fails, the 2.8% improvement cannot be attributed to tone removal, and the paper would need major revision.","tokens_in":10400,"tokens_out":7226,"duration_ms":63395,"concrete_test":"Randomly sample 200 questions from NQ-FS. For each, take the original factually accurate passage and the emotion-translator's neutralized output. Present both to three independent human annotators and ask whether every factual claim in the original is present and not contradicted in the neutralized version (yes/no per passage). Compute the fraction of passages judged fully fact-preserving and report inter-annotator agreement. Run the same protocol on the zero-shot neutralizer's outputs as a control. If the fine-tuned translator's preservation rate is not near-certain (e.g., at least 95%) or is not clearly above the zero-shot control, the Table 1 gain is confounded with factual editing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6 states that translating sarcastic text to neutral 'would effect the emotion of the text but not the factual content,' yet Section 6.2 only supports this via round-trip translations of human-written GoEmotions/SARC text. The one-way neutralization applied to NQ-FS, NQ-PSM, and NQ-PSA is never directly checked for fact preservation. Table 1 cannot establish it: with 10 retrieved passages, a corrupted neutralized passage can be masked by other passages or by the LLM's parametric knowledge, so QA accuracy is a weak proxy. If the one-way translator drops, adds, or flips facts, then the reported 2.8% gain on NQ-FS is not a sarcasm-comprehension effect but a content-editing artifact, and the claimed restoration to original-NQ performance would not generalize to other sarcastic corpora. The example in Table 3 even shows a round-trip where the sarcastic implication reverses ('only $1999.99' becomes 'a steal ... a bargain'), illustrating that semantic fidelity is not guaranteed. Additionally, the 'restores to original NQ' wording holds for the two Llama2 models but not for Qwen2-7B (43.5 vs. 44.1) or Qwen2-72B (48.9 vs. 49.2).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the Reading with Intent task, which studies how emotional tones in retrieved passages affect LLM question answering, from a single sarcasm trope to 11 emotions. The authors construct a large synthetic dataset by transforming Wikipedia-derived NQ retrieval passages into 11 emotional styles using five different LLMs, analyze its distributional properties, and fine-tune Llama-3.1-8B-Instruct with LoRA on this bitext corpus to train an emotion-translator. They evaluate the translator via BLEU and Amazon Mechanical Turk round-trip translation on human-written GoEmotions and SARC texts, reporting that the fine-tuned model reconstructs emotions and factual content better than a zero-shot baseline. They then apply the translator to neutralize sarcastic contexts in the NQ-FS, NQ-PSM, and NQ-PSA datasets, reporting an average 2.8% accuracy improvement on the factually accurate NQ-FS set across four reader LLMs, while neutralization leaves NQ-PSM and NQ-PSA largely unchanged. The paper concludes that neutralization helps LLMs handle sarcastic-but-factual context but is not sufficient for factually distorted sarcasm.","tokens_in":10671,"tokens_out":2586,"duration_ms":24188,"significance":"If the central claim holds, the work is a useful contribution to robust RAG: internet-retrieved contexts are not Wikipedia-like in tone, and a dedicated emotion-translator that strips sarcasm while preserving facts would be a practical preprocessing tool. The strengths include the scale of the synthetic dataset (3.6M passages across 11 emotions), the multi-LLM generation strategy that demonstrably reduces distributional shift (Figure 2), and the human evaluation design that tests generalization to unseen emotions (embarrassment, nervousness, relief) on out-of-distribution human text. The paper also honestly reports the null results on NQ-PSM and NQ-PSA. The main risk is that the load-bearing fact-preservation property of the one-way neutralization step is not directly validated on the benchmark passages, which the authors themselves partially acknowledge; this is addressable with additional experiments.","major_comments":[{"comment":"The claim that neutralizing NQ-FS 'restores the performance to the model's performance on the original NQ dataset' is not supported for two of the four LLMs. Qwen2-7B scores 43.5% with emotion-translator neutralization versus 44.1% on original NQ, and Qwen2-72B scores 48.9% versus 49.2%. The 2.8% average improvement should be reported with the per-model deltas and with some measure of variance; as presented, there are no error bars, confidence intervals, or significance tests, and the reader cannot tell whether the improvement is stable across evaluation seeds or passage samples.","section":"Section 6.1, Table 1"},{"comment":"Factual preservation is validated only on round-trip translations of human-written GoEmotions/SARC text, not on the one-way neutralization of the NQ-FS, NQ-PSM, and NQ-PSA passages that are used in Section 6.1. The paper asserts in Section 6 that neutralization 'would effect the emotion of the text but not the factual content,' but the provided evidence does not directly test this for the benchmark passages. The QA accuracy in Table 1 is a weak proxy for fact preservation because with 10 retrieved passages, a corrupted neutralized passage can be masked by other passages or by the LLM's parametric knowledge. The example in Table 3 itself illustrates the risk: the sarcastic 'only $1999.99' becomes 'a steal ... a bargain' after round-trip, changing the evaluative implication. A direct fact-preservation audit of the one-way neutralized NQ passages (e.g., human or automated comparison of atomic facts against the original non-sarcastic passage) is needed to establish that the 2.8% gain is a sarcasm-comprehension effect rather than a content-editing artifact.","section":"Section 6.2, Figure 6 and Table 3"},{"comment":"The abstract's 'about 3%' headline improvement is misleading without qualification: the improvement is confined to the NQ-FS dataset, while NQ-PSM and NQ-PSA show roughly zero net change (-0.35% and +0.07%). The body text reports this correctly, but the abstract should state that neutralization helps only for factually accurate sarcastic contexts, or report the dataset-specific numbers. In addition, the claim that the fine-tuned emotion-translator 'boosts performance by 1.05%' across datasets and models is an average over heterogeneous conditions and is not accompanied by a variance estimate or a breakdown that would show whether the gain is consistent beyond the NQ-FS rows.","section":"Abstract and Section 6.1"}],"minor_comments":[{"comment":"There is a typo: 'the authors focused on a single linguistic tope' should be 'trope.'","section":"Section 1, Introduction"},{"comment":"'This translation would effect the emotion' should be 'affect'; similar wording appears in the Section 6.1 discussion.","section":"Section 6, first paragraph"},{"comment":"The evaluation section lists four questions at the start but then describes experiments organized around three criteria (back-translation reconstruction, emotion recognition, factual fidelity). The mapping between the four questions and the presented figures and tables is unclear; the authors should align them explicitly.","section":"Section 6.2"},{"comment":"The caption says 'Human evaluation of the factual reconstruction ... as compared to an un-fine-tuned LLM,' but the accompanying text notes that raters could also select 'both models preserve factual fidelity equally well'; the caption should mention the three-way choice.","section":"Section 6.2, Figure 6 caption"},{"comment":"The sentence '10,000 sentence with 10 parallel versions of each sentence were used' mixes singular and plural and should be rephrased; also specify whether the 10,000 sentences are sampled from the full 3.6M-passage corpus and how the 90%/10% self-mapping split is applied per sentence.","section":"Section 5, training details"},{"comment":"The BLEU table reports a single average score for the base model and the translator, but the text says the average is 'across all emotions in the Go Emotions dataset'; including per-emotion BLEU scores would make the comparison more informative, especially given the acknowledged limitations of BLEU for many-to-many style transfer.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a natural extension of the authors' prior Reading with Intent work and is within scope for a CL journal. The central idea is sound and the human evaluation is a genuine strength. My main concern is the mismatch between the strength of the claim (fact-preserving neutralization improves comprehension) and the evidence for fact preservation in the one-way setting. This is fixable within the manuscript's scope by adding a direct fact-preservation audit on the neutralized benchmark passages and by reporting variance or significance for the Table 1 results. I would also encourage the authors to make the dataset and trained translator publicly available, since the paper currently gives no artifact link; the reproducibility of the human evaluation would benefit from releasing the 1,350-sample evaluation set."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Off the record: this is a solid incremental extension of the Reading with Intent program, and the central effect is probably real. The thing to know before reading closely is that the 2.8% NQ-FS gain rests on an assumption the paper never directly tests: that neutralizing sarcasm leaves the facts intact. Section 6.2 validates fact preservation on round-trip translations of human-written text, not on the one-way neutralization applied to the benchmark passages. The stress-test note is fair on that point. With only 10 retrieved passages and LLM parametric knowledge in the loop, QA accuracy is a weak proxy for fact preservation. The example in Table 3 shows how a round-trip can drift sarcasm from \"only $1999.99\" to \"a steal,\" so the concern is not hypothetical.\n\nWhat the paper does well: it builds a large 11-emotion synthetic dataset (3.6M passages) using five LLMs for generation, analyzes distributional properties, trains a LoRA fine-tuned translator, and validates it with human evaluation on external human-authored data. The translator generalizes to emotions not seen in training, which is real evidence the dataset captures something useful. The downstream neutralization effect is consistent across four LLMs, and the paper honestly reports that fact-distorted datasets show no improvement. That selectivity is actually a strength in the body, though the abstract's \"about 3%\" glosses over the null results.\n\nThe soft spots beyond fact preservation: no error bars or significance tests anywhere, so the 2.8% aggregate could be noise; the \"restores to original NQ\" claim holds for only two of four models; and no code, data, or prompts are released, which limits reproducibility. These are addressable, not fatal.\n\nThis paper deserves a serious referee. The central contribution is a new dataset and a trained translator that could be useful beyond this task. A reviewer should ask for a direct fact-preservation audit of the neutralized benchmark passages, significance testing, and data/code release. I would cite the dataset if it becomes available, and I would bring it to a reading group for the synthetic-data methodology alone.","headline":"A useful, honest extension of the Reading with Intent line whose headline effect is plausible but not yet proven, because fact preservation in the one-way neutralization step is never directly checked.","tokens_in":11215,"tokens_out":1460,"would_cite":true,"duration_ms":15956,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rewriting sarcastic context to neutral tone restores LLM question-answering accuracy to non-sarcastic levels.","keywords":["reading with intent","emotion translation","sarcasm neutralization","retrieval-augmented generation","question answering","style transfer","synthetic dataset","LoRA fine-tuning"],"falsifier":"Compare answer-bearing fact spans in the original NQ-FS sarcastic passages with their neutralized versions: if neutralization removes, adds, or reverses facts that determine the ground-truth answer, the reported comprehension gains are an artifact of changed content rather than tone. A concrete version is to run the same QA models on neutralized passages produced by a human editor with no knowledge of the question and check whether the 2.8 percent improvement survives.","tokens_in":10166,"feed_emoji":"🎭","tokens_out":4176,"duration_ms":33873,"temperature":0.7,"pith_summary":"The paper claims that the emotional tone of retrieved context, especially sarcasm, degrades LLM question answering, and that this degradation can be removed by rewriting the context to a neutral tone without changing its facts. To make that move work, the authors construct a synthetic parallel corpus in which Wikipedia-style passages are rewritten into eleven emotions by five different LLMs, then fine-tune a single 8-billion-parameter model with LoRA to translate any passage from one emotion to another. Applied to sarcastic reading-comprehension datasets, neutralizing the passages improves accuracy by about 2.8 percent on fully sarcastic context and restores performance to the non-sarcastic baseline. On datasets where some sarcastic passages are also factually distorted, neutralization leaves accuracy essentially unchanged, indicating that tone is only part of the problem. If correct, the work provides a cheap, model-agnostic preprocessing fix for one failure mode of retrieval-augmented generation on internet text.","feed_headline":"Neutralizing sarcasm restores LLM answer accuracy","feed_subtitle":"On fully sarcastic passages, rewriting to neutral tone recovers about 2.8 percent accuracy, matching non-sarcastic baselines.","key_machinery":"The central object is the emotion-translator: a Llama-3.1-8B-Instruct model fine-tuned with rank-8 LoRA on a synthetic parallel bitext where each passage is rewritten into eleven emotions or linguistic tropes: anger, condescension, disgust, envy, excitement, fear, happiness, humor, sadness, sarcasm, and surprise. Training pairs are prefixed with a prompt specifying the source emotion and the target emotion, and ten percent of examples map an emotion to itself as regularization. In the Reading with Intent pipeline, the translator takes sarcastic passages tagged by the intent-tagging system and rewrites them to neutral tone while retaining facts, and the downstream QA LLM then reads only the neutralized passages. The synthetic data-generation design, in which each passage is randomly assigned to one of five LLMs for each emotion, is what makes the parallel corpus diverse enough for the translator to generalize.","core_discovery":"Rewriting sarcastic retrieved context into neutral tone makes LLMs answer questions as accurately as when the context was never sarcastic. On the NQ-FS dataset, which contains fully sarcastic but factually accurate passages, neutralization improves QA accuracy by 2.8 percent on average across Llama-2-7B, Llama-2-70B, Qwen2-7B, and Qwen2-72B, and brings scores back to the original NQ no-sarcasm numbers. The same neutralization leaves NQ-PSM and NQ-PSA, which mix sarcasm with fact-distorted passages, nearly unchanged at -0.35 percent and +0.07 percent respectively, showing that sarcasm distorts comprehension through tone rather than through the deception signal carried by factual distortion. Human evaluations on round-trip translation of human-written emotional text from Go Emotions and SARC find that the fine-tuned translator preserves emotion and factual content better than a zero-shot LLM, including on emotions not present in the synthetic training set.","pith_inferences":["Neutralization could be deployed as a preprocessing layer in RAG systems that retrieve from social media, reviews, or forums where sarcasm is frequent; the paper does not test this directly but its datasets imitate that setting.","The roughly one percent additional gain from the fine-tuned translator over zero-shot neutralization suggests that task-specific fine-tuning on synthetic emotional bitext is worth the cost for tone normalization.","A natural extension is to make neutralization conditional on the downstream question, or to preserve emotional intensity while changing emotion type, which the paper's limitation section notes is not currently controlled.","The unchanged performance on fact-distorted sarcastic passages hints that sarcasm can serve as a weak deception signal; a testable follow-up is whether preserving a marker of possible deception while removing tone improves robustness."],"forward_implications":["Neutralizing tone before QA can recover essentially all of the accuracy lost to sarcastic context, without retraining the downstream model.","The approach degrades gracefully when sarcasm coincides with factual distortion, implying that neutralization should be paired with fact-checking or trust signals rather than used alone.","Round-trip translation with human evaluation offers a template for validating style and emotion transformations when parallel ground truth is unavailable.","Because the translator generalizes to three held-out emotions, embarrassment, nervousness, and relief, emotion translation may be trainable from a modest set of categorical emotions."],"supporting_citations":[{"why":"Supplies the Reading with Intent task, the NQ-FS, NQ-PSM, and NQ-PSA datasets, the Reading with Intent prompt, and the intent-tagging system that the emotion-translator is added to.","marker":"(Reichman et al., 2024)"},{"why":"Provides Natural Questions, the open-domain QA benchmark whose queries and Wikipedia passages form the base of both the new synthetic emotion dataset and the Reading with Intent evaluation.","marker":"(Kwiatkowski et al., 2019)"},{"why":"GPL retrieval algorithm used to select top-10 passages per NQ query for building the synthetic emotion dataset.","marker":"(Wang et al., 2021)"},{"why":"Frames retrieval-augmented generation, the setting in which retrieved context tone affects downstream answer quality.","marker":"(Lewis et al., 2020)"},{"why":"Provides the SARC dataset, one of the two human-written emotional-text sources used in the round-trip human evaluation.","marker":"(Khodak et al., 2018)"}],"fun_headline_variants":["Neutralize sarcasm, reclaim LLM answer accuracy","Rewriting sarcastic context to neutral lifts QA by ~3%","Sarcasm-neutralized context restores LLM accuracy","Tone-shifting sarcasm to neutral fixes RAG answers","Defuse sarcasm in retrieved text, improve LLM responses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The neutralization step keeps the facts of sarcastic passages intact; this was checked on human-written text via round-trip translation, not on the one-way neutralized NQ-FS, NQ-PSM, and NQ-PSA passages used in the accuracy numbers.","fun_headline_variants_meta":{"raw":{"variants":["Neutralize sarcasm, reclaim LLM answer accuracy","Rewriting sarcastic context to neutral lifts QA by ~3%","Sarcasm-neutralized context restores LLM accuracy","Tone-shifting sarcasm to neutral fixes RAG answers","Defuse sarcasm in retrieved text, improve LLM responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1403,"prompt_tokens":963,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":579,"tokens_out":440,"duration_ms":4562,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:51:53.732427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare answer-bearing fact spans in the original NQ-FS sarcastic passages with their neutralized versions: if neutralization removes, adds, or reverses facts that determine the ground-truth answer, the reported comprehension gains are an artifact of changed content rather than tone. A concrete version is to run the same QA models on neutralized passages produced by a human editor with no knowledge of the question and check whether the 2.8 percent improvement survives.","supporting_citations":[],"review_version":1}