{"id":"df82f30a-b6ac-40f5-902f-7e684d0adda4","arxiv_id":"2608.06377","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper presents MIST, a four-condition benchmark for measuring language models' selective trust of context, and SCOPE, a DPO-based training method that reduces misleading-signal susceptibility without hurting clean, correct-context, or irrelevant-context accuracy.","lead":"This paper introduces a benchmark that tests whether language models reject misleading hints while still using helpful ones, plus a training method that reduces how often wrong hints flip correct answers to wrong. It matters because deployed AI systems increasingly rely on external context, and a model that over-trusts or over-ignores context is unreliable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The clean-answerable MIST design limits 'benefit from correct context' to redundant-hint alignment, and external transfer never rewards helpful context; the general 'selective trust' claim is broader than the evidence, even though the reported correct>clean gaps and Standard-DPO collapse refute…","rationale":"The reader's conditional verdict is appropriate, but the stated weakest assumption needs correction. The data do not support the claim that a context-invariant model would match or beat SCOPE's reported profile: SCOPE has correct-context accuracy 3.1 and 8.0 points above its clean accuracy on Qwen3-4B and Llama-3.2-3B, respectively, so it is measurably using correct context. The Standard-DPO baseline on Llama also collapses correct-context accuracy to 56.4, demonstrating that the correct-context condition carries signal and that the balanced construction is doing real work. The load-bearing residual issue is one of scope rather than identifiability: because every clean item is answerable on its own, MIST only operationalizes 'benefit from correct context' as agreeing with a redundant hint, and the external transfer tasks never reward using beneficial context at all. Thus the paper robustly establishes balanced robustness to answer-like signals, while the broader 'selective trust' claim—especially the component about using context that is worth trusting because it supplies new information—is not yet tested. This is fixable by adding a context-necessary held-out condition or by weakening the abstract and conclusion. The benchmark, metric, and method remain publishable contributions, so the verdict stays conditional on that scope adjustment.","tokens_in":31412,"tokens_out":16769,"duration_ms":167843,"concrete_test":"Build or mark a held-out MIST-style subset whose clean condition is unanswerable or ambiguous without the supplied context (e.g., 'According to the memo, what is the reimbursement cap?' with no memo in the clean prompt), keeping matched misleading/correct/irrelevant variants. Evaluate SCOPE and a context-invariant control (same model prompted to ignore appended context) on this subset. If SCOPE's correct-context accuracy does not exceed the context-invariant control's, or if it does not exceed its own clean accuracy on those items, then the 'benefit from correct context' component of selective trust is unsupported. Running this on Qwen3-4B and Llama-3.2-3B with the released checkpoints would settle the scope of the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's specific context-invariance objection is partly refuted by the paper's own table: a model that ignored all context would have equal accuracy across the four conditions, whereas SCOPE's correct-context accuracy exceeds its clean accuracy on both trainable families (Qwen3-4B: 98.1 vs 95.0; Llama-3.2-3B: 80.0 vs 72.0), and Standard-DPO's collapse on correct-context (56.4 vs 80.0 for SCOPE on Llama) shows the correct-context condition is informative. The residual load-bearing concern is that Appl. A.4 requires every clean item to be 'answerable on its own,' so correct-context and irrelevant-context rows only add answer-redundant hints. Consequently MIST measures whether a model keeps a clean-solvable answer when a hint agrees, disagrees, or is neutral; it never tests whether the model uses correct context to supply information absent from the question. The external transfer tasks (GSM-IC, GSM-Plus, Sharma) also never reward using a beneficial context. Since the introduction defines selective trust as including 'benefit from context that is actually correct,' the evidence supports 'balanced robustness to answer-like signals' and 'selective trust over redundant hints,' but not the broader claim that SCOPE learns to trust context that supplies new information. This matters because the paper's motivation explicitly rejects context-invariance as useless when context is worth trusting.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIST, a human-annotated benchmark of 1,000 reasoning items each rendered under four matched conditions (clean, misleading, correct-context, irrelevant-context), together with SC2W, a paired metric that measures how often a misleading signal flips a clean-correct answer to wrong. Using MIST, the authors document substantial susceptibility to misleading signals across 23 API and open-weight models. They then propose SCOPE, which constructs matched preference pairs from a base model's clean-correct/misleading-wrong failures and applies a standard DPO objective with equal sampling across the four conditions. Experiments on two trainable families (Qwen3-4B and Llama-3.2-3B) show large SC2W reductions while preserving or improving the three control accuracies, with construction ablations and zero-shot transfer to GSM-IC, GSM-Plus, and Sharma-style sycophancy tasks.","tokens_in":31629,"tokens_out":11008,"duration_ms":102769,"significance":"The empirical work is careful and well executed: training and evaluation pools are item-disjoint, answer scoring is deterministic exact match, the matched-condition design controls for task difficulty, and the ablations isolate the contribution of matched pairing and the control components. The benchmark and SC2W metric are potentially useful community resources, and the reported robustness gains on two model families are credible. However, the central conceptual claim that SCOPE learns 'selective trust'—defined in the introduction as including the ability to 'benefit from context that is actually correct'—is broader than what the evaluation can support. Because the clean condition is answerable on its own, the correct-context rows add only answer-redundant hints, and none of the external transfer tasks rewards using a beneficial context. The evidence supports a claim of balanced robustness to misleading and irrelevant signals with preserved clean reasoning, but not the stronger claim that the model learns to trust context that supplies genuinely new information.","major_comments":[{"comment":"The central claim that SCOPE learns 'selective trust,' defined in the introduction as including 'benefit from context that is actually correct,' is not supported by the reported evaluation. In Appl. A.4 the clean condition is required to be 'answerable on its own,' so the correct-context rows in MIST add only answer-redundant hints; the benchmark never presents a case where the correct context supplies information not already available in the question. Moreover, in Eq. (5) the chosen response r+ for the correct-context pair is the same context-free clean-condition response used under the other conditions, so the DPO objective for D_cor rewards producing the clean answer under a correct-context prompt rather than rewarding use of the context. Consequently the observed correct>clean gaps (e.g., Qwen3-4B: 98.1 vs 95.0; Llama-3.2-3B: 80.0 vs 72.0 in Table 1) rule out pure context-invariance but demonstrate only that the model benefits from redundant hints; they cannot distinguish 'selective trust' from a heuristic that trusts context only when it confirms the model's own tentative answer. Please either add a condition or a separate task in which the correct answer is only derivable from the provided context and show that SCOPE improves there, or reframe the paper's claim to 'balanced robustness to misleading signals with preserved accuracy on clean, correct, and irrelevant contexts.'","section":"§4, Eq. (5); §3.1, Appl. A.4"},{"comment":"The external-transfer claim that 'the learned behavior transfers zero-shot' is presented as evidence that SCOPE 'transfers by learning when to rely on context, not by ignoring it.' However, none of the three external suites rewards using a beneficial context: GSM-IC adds irrelevant distractors, GSM-Plus perturbs problem statements, and Sharma tests sycophancy. These results therefore support the transfer of robustness to misleading and irrelevant signals, but not the transfer of selective use of helpful context. The phrasing in §5.4 goes beyond what the metrics can distinguish; please temper the claim or add an external task with a helpful-context condition.","section":"§5.4, Table 2"}],"minor_comments":[{"comment":"The panel reports 188 rescued (56.8%) out of 331 vulnerable items and also a net of +182 repairs with 200 gains and 18 costs; the relationship between these counts (for example, why repairs are fewer than gains) is not explained in the text.","section":"Fig. 7(a)"},{"comment":"The SC2W human-audit Spearman correlation is 0.82 with a 95% bootstrap CI of [0.30, 1.00] based on only six systems, so the claim that 'agreement is strong across all six metrics' is too strong for SC2W; please temper the wording or report the interval alongside the strength claim.","section":"§5.6, Table 5"},{"comment":"The statement that SCOPE 'roughly halves SC2W' is accurate for Qwen3-4B (35.0 to 16.3) but not for Llama-3.2-3B (31.5 to 20.6); consider replacing 'roughly halves' with 'substantially reduces.'","section":"Conclusion"},{"comment":"The claim 'a single misleading signal costs 17.1 points on average' should define the underlying quantity at the point of use, since the body defines only SC2W and the conclusion appears to refer to the clean-minus-misleading accuracy gap rather than SC2W itself.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is solid and the resource contributions (MIST, SC2W, and the balanced preference construction) are likely to be useful. The main weakness is that the conceptual framing of 'selective trust' overreaches the evidence: the benchmark and transfer tasks never test whether the model benefits from context that provides information unavailable in the question, and the DPO construction for the correct-context condition actually rewards a context-free answer. This is fixable by adding a necessary-context evaluation or by softening the central claim, but it is load-bearing for the paper's motivation and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a useful, carefully executed empirical paper. MIST is a genuinely new artifact, SC2W is a sensible paired metric, and SCOPE's balanced DPO data construction is simple enough to be adopted elsewhere. The experiments are honestly run: deterministic exact-match scoring, item-disjoint train/test, ablations that isolate the matched-pairing and control components, and zero-shot transfer checked on three external suites. I also appreciate that the paper says outright that the novelty is in the data, not the optimizer.\n\nThe main soft spot is the central claim. The stress-test note is right that a pure context-invariance story is refuted by the paper's own numbers: SCOPE's correct-context accuracy exceeds clean accuracy on both trainable families (98.1 vs 95.0 on Qwen3-4B; 80.0 vs 72.0 on Llama-3.2-3B), and Standard-DPO's collapse on correct-context (56.4 vs 80.0) proves that condition is not vacuous. So the mechanism is not 'ignore everything.' But the residual concern still bites. Because MIST's clean condition is answerable on its own, the correct-context condition only ever supplies a redundant hint. Neither MIST nor the external transfer tasks (GSM-IC, GSM-Plus, Sharma) ever require the model to use correct context to obtain information that is absent from the question. The evidence supports 'balanced robustness to answer-like signals' and 'selective trust over redundant hints'; it does not support the broader reading of 'benefit from context that is actually correct' that the introduction and conclusion reach. This is an overclaim, not a fatal flaw, and it is fixable by adding a condition where correct context is necessary or by carefully weakening the wording.\n\nMinor points: only two trainable families, both smallish (3B/4B), so cross-family claims should be read cautiously. The SC2W human audit has a wide confidence interval (Spearman 0.82, CI [0.30, 1.00]), though it is only a scoring sanity check. The 'universal susceptibility' claim rests on one benchmark, but the 23-model spread makes it plausible.\n\nThe paper is clearly written, the limitations section is candid, and the benchmark plus metrics are likely to be reused. This deserves a serious referee. I would send it to peer review and ask for a revised framing of what selective trust means empirically, rather than rejecting it.","headline":"MIST/SC2W and the SCOPE recipe are solid, reproducible contributions, but the 'selective trust' headline runs ahead of the evidence since correct context never supplies information the question lacks.","tokens_in":32273,"tokens_out":2527,"would_cite":true,"duration_ms":25352,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that language models should be trained to judge external context on its merits, and that a balanced preference-optimization scheme roughly halves the rate at which misleading hints flip correct answers to wrong.","keywords":["selective trust","misleading context","Direct Preference Optimization","SC2W","MIST benchmark","context robustness","language model alignment","sycophancy"],"falsifier":"Construct a matched item set in which the clean condition is unanswerable without the correct-context passage, and compare a SCOPE-trained model against the base model and a context-ignoring baseline on those items; if the SCOPE model shows no accuracy gain over ignoring context, the claim that it learns selective trust is falsified.","tokens_in":31125,"feed_emoji":"🧭","tokens_out":8240,"duration_ms":77063,"temperature":0.7,"pith_summary":"The paper argues that resisting misleading context is the wrong training target, because a model that ignores all outside signals looks robust in tests that only compare clean prompts with prompts carrying wrong hints, yet is useless when the context is worth trusting. It recasts the problem as selective trust and introduces MIST, a human-annotated benchmark that renders each reasoning item under four matched conditions—clean, misleading, correct-context, and irrelevant-context—so that only the added signal changes. Across 23 models the authors report that a single misleading signal costs 17.1 points of accuracy on average. They then propose SCOPE, which mines cases where a base model answers correctly on its own but wrong under a misleading signal, and trains a standard DPO objective over matched preference pairs balanced equally across all four conditions. The paper's central claim is that this markedly lowers the signal-induced correct-to-wrong rate (SC2W) on two open-weight model families while preserving accuracy on clean, correct-context, and irrelevant-context inputs, and that the behavior transfers zero-shot to external benchmarks.","feed_headline":"Balanced preference data halve misleading-context flips in LLMs","feed_subtitle":"One misleading hint costs 17.1 accuracy points on average; a matched-context benchmark shows balanced DPO recovers most.","key_machinery":"The load-bearing object is the matched four-condition item design and the paired SC2W metric, together with the balanced preference construction. Each MIST item holds question, answer space, gold answer, and plausible wrong answer fixed while varying only the added context: none (clean), a signal toward a wrong answer (misleading), a signal toward the gold answer (correct), and non-answer-bearing text (irrelevant). SC2W is the fraction of items the model answers correctly under clean but wrongly under misleading, isolating signal-induced flips from raw ability. SCOPE reuses one chosen/rejected response pair across all four context prompts, samples the four conditions with equal 25% mass, and optimizes the standard full-completion DPO loss over the combined objective; the construction ablation is what the paper uses to argue that matched pairing and control balance are what matter rather than the loss function.","core_discovery":"The central discovery is that susceptibility to misleading context is universal and separable from task ability: SC2W conditions on items the model already solves when clean, and across 23 frontier and open-weight models a misleading signal flips these clean-correct answers to wrong 17.1 points on average. The paper further claims that resistance training is not selective trust: prompt-defense, supervised fine-tuning, and misleading-only DPO all raise misleading-context accuracy while eroding clean or correct-context accuracy. SCOPE keeps the standard full-completion DPO loss unchanged and changes only what enters it—matched signal-counterfactual pairs balanced equally across misleading, clean, correct-context, and irrelevant-context conditions. On Qwen3-4B it cuts SC2W from 35.0 to 16.3, and on Llama-3.2-3B from 31.5 to 20.6, while preserving or improving the three control accuracies; the learned behavior is claimed to transfer zero-shot to GSM-IC, GSM-Plus, and Sharma-style sycophancy evaluation.","pith_inferences":["Inference: the reported metrics are also consistent with a model that simply ignores all added context, because MIST's clean questions are answerable on their own and the correct-context condition never requires using the context to improve on clean reasoning.","Inference: the external transfer tasks (GSM-IC, GSM-Plus, sycophancy-style items) likewise never reward using a helpful context, so zero-shot transfer does not by itself demonstrate benefit-from-correct-context behavior.","Inference: a direct test would add a condition in which the clean prompt cannot be answered without the correct context; if SCOPE-trained models improve on such items relative to base, selective trust is real, and if not, the method has produced balanced context-invariance."],"forward_implications":["Context-robustness evaluation should use matched multi-condition designs, since single-condition tests cannot distinguish a model that distrusts all context from one that judges signals correctly.","Training with balanced control pairs should roughly halve SC2W on open-weight families: the paper reports reductions from 35.0 to 16.3 on Qwen3-4B and from 31.5 to 20.6 on Llama-3.2-3B.","Resistance-only baselines are expected to trade correct-context accuracy for misleading-context accuracy; the paper reports Standard-DPO collapsing correct-context accuracy on Llama-3.2-3B.","The learned behavior is claimed to transfer zero-shot to GSM-IC, GSM-Plus, and sycophancy-style items, meaning the balanced preference data teaches a general stance rather than a prompt template."],"supporting_citations":[{"why":"supplies the standard full-completion DPO objective that SCOPE keeps fixed while changing only the preference data.","marker":"Rafailov et al., 2023"},{"why":"documents that answer-like cues steer model predictions without surfacing in stated reasoning, motivating the SC2W flip metric.","marker":"Turpin et al., 2023"},{"why":"the closest resistance-training neighbor, whose synthetic-data approach SCOPE contrasts with by adding matched control conditions.","marker":"Wei et al., 2023"},{"why":"GSM-IC, used as a zero-shot external transfer test for distraction by irrelevant context.","marker":"Shi et al., 2023"},{"why":"GSM-Plus, used as a zero-shot external transfer test for robustness to perturbed problem statements.","marker":"Li et al., 2024"},{"why":"sycophancy examples and the finding that preference alignment can reward agreement with user beliefs, used as an external transfer test and motivation.","marker":"Sharma et al., 2024"}],"fun_headline_variants":["Balanced pairs halve misleading-context flips","SCOPE: balanced DPO cuts context flips in LLMs","Selective trust beats ignore-all-context training","Matched preference data halve LLM flips on lies","Balanced DPO slashes misleading-hint flips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark's correct-context condition proves the model benefits from correct context; because every clean question is answerable on its own, a model that ignores all added context would score well on every reported condition, so the numbers alone do not distinguish selective trust from balanced context-invariance.","fun_headline_variants_meta":{"raw":{"variants":["Balanced pairs halve misleading-context flips","SCOPE: balanced DPO cuts context flips in LLMs","Selective trust beats ignore-all-context training","Matched preference data halve LLM flips on lies","Balanced DPO slashes misleading-hint flips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1332,"prompt_tokens":970,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":280}},"tokens_in":586,"tokens_out":362,"duration_ms":4609,"temperature":1.0,"reasoning_tokens":280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:03:23.818500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a matched item set in which the clean condition is unanswerable without the correct-context passage, and compare a SCOPE-trained model against the base model and a context-ignoring baseline on those items; if the SCOPE model shows no accuracy gain over ignoring context, the claim that it learns selective trust is falsified.","supporting_citations":[],"review_version":1}