{"id":"6ec70cc0-581f-4d15-ab24-3c1fa7a341d4","arxiv_id":"2507.01543","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"External information from Wikipedia or web search degrades LLM stance detection performance in most configurations, mainly because models align predictions with the stance of the added text.","lead":"This paper tests whether giving large language models extra background information, like Wikipedia excerpts, helps them decide if a tweet supports, opposes, or is neutral toward a target. Across eight models and three datasets, the extra information usually hurts accuracy, sometimes sharply, because models tend to adopt the stance of the added text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-information baseline omits the entire instruction sentence that introduces external information, so the observed degradation is not isolated to information content.","rationale":"After parsing the full text, including Appendix B prompts and Section 4, the most load-bearing threat to the central claim is the prompt-structure confound identified by the reader. The no-information baseline omits the entire sentence used to present external information; in the CoT variants, it also omits the directive not to adopt the information's stance. Since the paper's headline is about the content of external information, not about prompt formatting, this confound directly undercuts the causal reading of Table 1. The paper's own length-correlation analysis does not address presence, and the NAR/HAR analysis assumes content is what moves predictions. I do not see a stronger objection: other limitations, such as single runs, absent error bars, and the headline 27.9% figure relying on synthetic biased information, are real but secondary, and the paper is transparent about its setup and releases code. The random-text control proposed would settle whether content or structure drives the degradation. Given that this check has not been run, the CONDITIONAL verdict is appropriate, and no verdict change is needed.","tokens_in":11813,"tokens_out":4099,"duration_ms":46811,"concrete_test":"Run the main non-CoT and zero-shot CoT evaluations with a random-text control condition: keep the prompt exactly as in the external-information condition, but replace {wiki} with a length-matched, stance-neutral filler text (e.g., a Wikipedia article on an unrelated topic or repeated lorem ipsum), using the same four representative models, Llama-3.1-8B, Qwen2.5-7B, GPT-4o mini, and Claude 3 Haiku, across all 12 targets. Compare macro F1 of this condition with the no-info baseline and with the real external-information conditions. If the random-text condition reproduces a substantial fraction, e.g., within statistical error or more than 50%, of the degradation observed with real information, the central claim that external information content harms performance fails; if random text is indistinguishable from no-info while real information degrades, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix B shows that the central comparison is confounded. In the non-CoT prompt, the condition with external information inserts the sentence \"The following information can be helpful: {wiki}.\" That sentence is omitted entirely in the no-information condition. In the zero-shot and few-shot CoT prompts, the inserted sentence is \"Integrate the following external information and do NOT automatically adopt the stance of it: {wiki}.\" Here, too, the whole sentence, including the directive \"do NOT automatically adopt the stance,\" is absent from the no-information baseline. Every reported drop in Table 1 therefore conflates two changes: (1) semantic content of the Wikipedia/web text, and (2) a prompt-structure change, namely an extra instruction sentence, an explicit instruction about not adopting stance, and greater prompt length. The NAR/HAR analysis in Section 4.2 attributes prediction shifts to the stance or sentiment perceived in the information, but it cannot separate drawing on content from reacting to the added instruction. The length correlation in Figure 2 is not a control for this: every point in that figure already has the extra sentence present, so length does not test presence. Because the headline claim is specifically that external information content degrades LLM stance detection, the semantically empty prompt-structure alternative must be ruled out before the claim is secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether adding external information (Wikipedia excerpts or web search results) to LLM prompts improves stance detection. Across eight LLMs, three datasets, and twelve targets, the authors report that external information degrades performance in most configurations, with macro-F1 drops up to 27.9%, in contrast to a fine-tuned BERT baseline (WS-BERT) that generally improves. They propose net adoption rate (NAR) and harmful adoption rate (HAR) metrics to show that models align predictions with the stance or sentiment of the provided information, and they test chain-of-thought prompting and LoRA fine-tuning as mitigations. The paper concludes that external information often harms LLM stance detection and should be used with caution.","tokens_in":12039,"tokens_out":4690,"duration_ms":56288,"significance":"If the central claim is secure, this is a useful cautionary result for the growing practice of augmenting LLM prompts with retrieved text: it would show that knowledge injection, which helps fine-tuned BERT-style models, can backfire for prompted LLMs. The study's strengths are its breadth (eight models, three datasets, multiple information sources, CoT variations, and fine-tuning), the release of code, and the introduction of NAR/HAR as interpretable diagnostics. However, the headline conclusion is not yet fully supported because the no-information baseline differs from the information condition in prompt structure, and because single-run results lack uncertainty quantification. With additional control experiments and statistical rigor, this could be a strong contribution to LLM robustness and stance detection research.","major_comments":[{"comment":"The central comparison is confounded. In the non-CoT prompt, the information condition adds the sentence \"The following information can be helpful: {wiki}.\" and the no-information condition omits this sentence entirely. In the zero-shot and few-shot CoT prompts, the information condition adds \"Integrate the following external information and do NOT automatically adopt the stance of it: {wiki}.\", and again this whole sentence is absent in the no-information baseline. Every performance difference in Table 1 therefore conflates the semantic content of the external information with a prompt-structure change: an extra instruction sentence, an explicit directive about stance adoption, and greater prompt length. The abstract and Section 4.1 attribute the degradation to \"such information\" / \"external information,\" which is a causal claim about content. This cannot be sustained without a control condition that matches the prompt structure while varying only the content—for example, inserting a random or irrelevant text of similar length, or explicitly stating \"No additional information is provided.\" The length-correlation analysis in Figure 2 does not resolve this, since all points in that figure already have the extra sentence present; length is not presence. This issue must be fixed or the claims rephrased as \"adding an instruction sentence together with external information degrades performance.\"","section":"Appendix B / Section 4.1"},{"comment":"The paper reports single runs with no variance estimates or significance tests. Section 3.1 justifies single runs by temperature zero producing \"negligible variation in performance,\" but this does not eliminate sampling variability in the test set or sensitivity to prompt and decoding details, and API models can be nondeterministic. Many of the reported changes are small (e.g., -0.2% accuracy for Llama-3.1-8B on COVID-19-Stance with Wikipedia, Table 1), so without standard errors or bootstrap confidence intervals the sign of the change is not reliable. The aggregate claim \"external information decreases performance in most cases\" is literally a count of negative cells in Table 1, so the absence of uncertainty quantification is load-bearing. Please provide repeated runs or resampling-based confidence intervals, at least for the main no-info-vs-Wikipedia comparison, and report how many individual differences are statistically distinguishable from zero.","section":"Section 3.1 / Table 1"},{"comment":"The NAR/HAR analysis and the length correlation inherit the confound described in the first major comment. The definition of NAR compares M(x|t,T) with M(x|∅,T), where the latter omits the entire instruction sentence; the observed positive net adoption could therefore reflect the model reacting to the added sentence rather than to the stance or sentiment of the information content. Figure 2 correlates length with prediction rates and relative performance only within information-present conditions, so it does not test whether the presence of the extra sentence is the cause. Additionally, s = M(t|∅,T) is the model's stance or sentiment of the information obtained through a separate, simpler prompt (Appendix B), which is a different task from stance detection of the tweet; the paper does not validate that this isolated perception matches how the model internally represents the information when it is embedded in the detection prompt. Please either add the missing control or explicitly limit the NAR/HAR claims to \"under this prompt structure,\" and consider validating the perceived-stance prompt against a fixed external sentiment model.","section":"Section 4.2, Eq. (1), and Figure 2"}],"minor_comments":[{"comment":"The omitted sentences are marked in red, which is not visible in a grayscale printout; please use a clearer notation such as brackets or a footnote indicator.","section":"Appendix B"},{"comment":"The claim of \"negligible variation in performance\" should be quantified; if possible, report the observed variance from a few pilot runs or a range rather than an informal assertion.","section":"Section 3.1"},{"comment":"The p-values are reported without multiple-comparison correction; with four correlations, the one significant result (p=0.004) should be interpreted cautiously, and the text should note this.","section":"Figure 2"},{"comment":"The fine-tuning analysis reports counts of instances with negative relative performance but does not distinguish large negative changes from small ones near zero; adding a magnitude summary (as in Table A3) in the main text would help the reader judge the practical relevance.","section":"Section 4.4 / Table A2"},{"comment":"The external information for SemEval 2016 Task 6 was collected by the authors via the Wikipedia API; a brief description of the extraction process and any manual quality checks would improve reproducibility.","section":"Section 3.2"},{"comment":"The Limitations section does not mention the prompt-structure difference between the no-info and info conditions; even if the control experiments are added, this limitation should be discussed explicitly.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The empirical scope is impressive and the paper addresses a timely question, but the central attribution is currently underdetermined by the prompt confound. I would like to see the authors add a content-control condition (e.g., random-text or irrelevant-text insertion of comparable length) and show that the degradation persists or disappears accordingly. If the degradation survives the control, the paper will be a strong candidate; if it disappears, the paper's conclusion would need to be reframed as a prompt-engineering effect. The single-run issue is also important because many differences are a few percentage points and the aggregate claim rests on their sign. This is fixable within a revision, so I do not recommend rejection, but the current version is not yet ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this much: it's a genuinely useful empirical study, but the headline result is not as clean as it looks. The comparison that drives the paper—external info present vs. absent—also changes the prompt: the no-info condition omits the sentence 'The following information can be helpful: {wiki}' (or the 'do NOT automatically adopt' variant in the CoT prompts). So the observed F1 drops could be caused by the extra instruction, not by the semantic content of the Wikipedia or web text. That's a real confound, and the paper does not address it.\n\nWhat is good: the scale. Eight LLMs, three datasets, twelve targets, four information types, plus CoT and LoRA experiments. The contrast with a fine-tuned BERT baseline is a nice control showing the effect is specific to these models. The NAR/HAR analysis is a sensible attempt to show that models move their predictions toward the stance/sentiment of the supplied text. The paper is honest about limitations and the code is released. The 'degrades in most cases' framing is accurate: the drops are consistent in direction if not always large.\n\nThe soft spots beyond the confound: single runs with no variance. Setting temperature to zero reduces stochasticity, but tokenizer/ordering effects and API nondeterminism can still produce spread; a few seeds would make the table sturdier. Also, the length-correlation analysis in Figure 2 doesn't rescue the main claim because length is not presence—every point in that figure already has the instruction sentence. A random-text control (e.g., an irrelevant Wikipedia passage or shuffled tokens) would isolate content from prompt form. Without it, the paper can still say 'external information as presented here' hurts, but it cannot say 'the information content' is the culprit.\n\nI would not reject this; I would send it back for a reasonable revision. The empirical sweep is too wide to ignore, and the negative result, if it holds up under a proper control, matters for anyone building RAG-based classifiers. The fix is cheap. A serious referee should look at it, and the authors should be given the chance to lock the claim down.","headline":"A wide, honest empirical study whose headline 'external info hurts' is entangled with a prompt-structure difference; fixable and worth sending to review.","tokens_in":12547,"tokens_out":2464,"would_cite":true,"duration_ms":28827,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"External information from Wikipedia or the web usually hurts LLM stance detection, cutting macro F1 by up to 27.9 percent.","keywords":["stance detection","large language models","external information","Wikipedia excerpts","web search","information bias","chain-of-thought prompting","macro F1"],"falsifier":"A random-text control condition would settle the claim: present the same prompt with length-matched but stance-free filler text instead of Wikipedia or web-search excerpts, keeping the instruction sentence present in every condition. If macro F1 falls as much with filler as with real external information, then the degradation is a prompt-structure artifact, not evidence that external information biases LLM predictions.","tokens_in":11626,"feed_emoji":"📉","tokens_out":9568,"duration_ms":95128,"temperature":0.7,"pith_summary":"This paper asks whether giving a large language model background information about a target—Wikipedia excerpts or web-search summaries—makes its stance detection better. Across eight LLMs and three tweet datasets, the answer is usually no: adding such information lowers macro F1 in most model–dataset combinations, with the largest drop at 27.9%. The paper traces the harm to a mechanism: LLMs align their prediction with the stance or sentiment they perceive in the supplied information rather than with the ground-truth stance of the text. Chain-of-thought prompting does not remove the effect, and fine-tuning reduces but does not fully eliminate it. A fine-tuned BERT baseline improves with the same information, so the finding is specific to how LLMs use context.","feed_headline":"External info hurts LLM stance detection, by up to 27.9%","feed_subtitle":"Wikipedia and web-search excerpts bias LLMs toward the source's stance rather than the text's own stance.","key_machinery":"The analytic instruments are the net adoption rate (NAR) and the harmful adoption rate (HAR), defined in Equation 1. NAR measures whether a model is more likely to produce a prediction matching the stance or sentiment it perceives in the external information when that information is present than when it is absent, and HAR measures whether such adoptions turn correct predictions into incorrect ones more often than the reverse. These rates convert the intuition that the model is swayed by the supplied text into a measurable quantity, giving a causal account of the performance drops observed in the main comparison.","core_discovery":"The central discovery is that external information from Wikipedia or web search degrades stance detection performance of LLMs in most cases rather than helping it. The paper shows this through a systematic comparison of eight LLMs on three datasets with 12 targets, where macro F1 drops by up to 27.9% relative to the no-information baseline. The degradation is explained by the models' tendency to adopt the stance and sentiment they perceive in the supplied information: net adoption rates are positive for nearly all model–dataset combinations, and the adoptions are more often harmful than helpful. The paper further shows that chain-of-thought prompting does not eliminate the degradation and that fine-tuning only narrows it, in contrast to a BERT-based baseline that benefits from the same external information.","pith_inferences":["The same stance-adoption mechanism likely applies beyond three-way stance classification: any retrieved passage that carries a stance may shift an LLM's answer toward that stance.","A concrete extension would be to score each retrieved passage's stance and re-weight or filter passages before prompting; the paper does not test such a mitigation.","Taken further, a stance-tilted web source is not just a performance nuisance but a potential poisoning channel for deployed stance classifiers that consume external text."],"forward_implications":["In zero-shot use, Wikipedia and web-search excerpts should not be assumed to help LLM stance classifiers; in most model–dataset pairs they lower macro F1, so systems should be evaluated with and without them.","The bias mechanism predicts that source reliability matters: a neutral or counter-stanced excerpt will push predictions in its direction even when the text itself says otherwise.","Chain-of-thought prompting does not cure the bias, so instruction-based warnings not to adopt the information's stance are not an adequate safeguard.","Fine-tuning with external information shrinks but does not eliminate the degradation, so mitigation requires more than continued training on the same prompt distribution.","The contrast with the BERT-based baseline indicates that the degradation is not a property of the task or datasets but of how LLMs attend to supplied context."],"supporting_citations":[{"why":"Supplies the Wikipedia-based external-information setup and the WS-BERT baseline whose improvement is the direct contrast to LLM degradation.","marker":"He et al. (2022)"},{"why":"Provides the COVID-19-Stance dataset used for evaluation.","marker":"Glandt et al. (2021)"},{"why":"Provides the P-Stance dataset used for evaluation.","marker":"Li et al. (2021)"},{"why":"Provides the SemEval 2016 Task 6 dataset used for evaluation.","marker":"Mohammad et al. (2016)"},{"why":"Supplies the zero-shot chain-of-thought prompting method tested in RQ3.","marker":"Kojima et al. (2022)"},{"why":"Supplies the few-shot chain-of-thought prompting method tested in RQ3.","marker":"Wei et al. (2022b)"},{"why":"Supplies the LoRA fine-tuning method used in RQ4.","marker":"Hu et al. (2022)"},{"why":"Supports the length-correlation analysis showing that prompt length does not explain the performance changes.","marker":"Sclar et al. (2024)"},{"why":"Supplies the web search API used to collect the real-world web-search information condition.","marker":"OpenAI (2025a)"}],"fun_headline_variants":["External info hurts LLM stance detection by up to 27.9%","Wikipedia excerpts backfire, dropping LLM stance F1 by 27.9%","LLMs adopt external stance, harming detection by up to 27.9%","Unlike BERT, external info hurts LLM stance detection","External info biases LLMs, lowering stance F1 up to 27.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the measured degradation comes from the content of the external information and not from the extra instruction sentence that is added only when external information is present; if the added sentence alone causes the drop, the central claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["External info hurts LLM stance detection by up to 27.9%","Wikipedia excerpts backfire, dropping LLM stance F1 by 27.9%","LLMs adopt external stance, harming detection by up to 27.9%","Unlike BERT, external info hurts LLM stance detection","External info biases LLMs, lowering stance F1 up to 27.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001158,"raw_usage":{"total_tokens":4783,"prompt_tokens":917,"completion_tokens":3866,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":3764}},"tokens_in":533,"tokens_out":3866,"duration_ms":30353,"temperature":1.0,"reasoning_tokens":3764,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:48:08.085604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A random-text control condition would settle the claim: present the same prompt with length-matched but stance-free filler text instead of Wikipedia or web-search excerpts, keeping the instruction sentence present in every condition. If macro F1 falls as much with filler as with real external information, then the degradation is a prompt-structure artifact, not evidence that external information biases LLM predictions.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COVID-19-Stance dataset used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SemEval 2016 Task 6 dataset used for evaluation."}],"review_version":1}