Pith.

Artificial Intelligence · arXiv:2610.20805 v2 · submitted 14 Oct 2026

When Reasoning Traces Conflict with Evidence: How Answer Configuration Shapes Evidence Following

Shaoang Li, Yanhang Shi, Jian Li

The paper says

“A reasoning trace retained at answer time plays two roles: it can raise accuracy, and it can compete with the current evidence when the two disagree.”

more of the abstract

We disentangle these roles by crossing verified evidence versions with the traces generated from each and answering from the same traces and evidence under two configurations: continuation of the model's own thinking and a separate answer call. Across three model families (gpt-oss, Gemma4, Qwen3) on HotpotQA and MuSiQue, the separate call raises evidence-following over continuation by 29.9 to 60.6 points. On the same pairs, removing the trace lowers accuracy by similar amounts under both configurations. Yet continuation repeats the stale conclusion in most conflicts, and the separate call follows the current evidence far more often. Benefiting from a trace and deferring to it are therefore distinct properties. Further analyses show that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning, unlike a verification instruction, follows the current evidence in most conflicts. Because trace deletion alone cannot reveal which source an answer follows, evaluations should report both properties together with the answer configuration.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. A clean empirical dissociation between trace utility and conflict resolution, with large effect sizes; the paper belongs in a good venue despite acknowledged limits on generalizability. This line is wrong

2 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

2 minor objections

The paper says, at §2.1

Pith says: minor objection 1 of 2

The description of the 'separate answer call' would benefit from an explicit statement that the answer is generated with thinking turned off, to contrast directly with continuation where thinking remains active. This is implied by the prompt structure but could be made explicit when the configuration is first defined.

The paper says, at §2.4

Pith says: minor objection 2 of 2

The 12,000-character cut for MuSiQue contexts is mentioned only at the end of Section 2.4 and then again in Appendix B.2. Moving a brief justification or reference to the appendix earlier (e.g., to the main ablation description) would help a reader who does not consult the appendix immediately.

What would settle it

Replicate the crossed-conflict experiment for one model (e.g., Qwen3-14B on HotpotQA) using traces that have been automatically paraphrased to remove explicit answer statements while preserving the reasoning steps. If the evidence-following gap between continuation and separate call shrinks by more than 15 percentage points relative to the original (60.1−14.1=46.0), the dissociation may depend on trace explicitness, which the paper acknowledges as a limitation.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic partly agrees 57 s · 36,211 tokens read, 5,990 written
  3. Editor wrote the verdict above 10 s · 36,866 tokens read, 889 written
  4. Referee accept: 2 minor 16 s · 36,411 tokens read, 1,465 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does something straightforward that needed doing: it holds the trace fixed and asks whether benefiting from a reasoning trace and deferring to it when evidence changes are the same behavior. They aren't. Across three model families and two datasets, a separate answer call follows current evidence 30–60 points more often than continuation does, while trace removal costs both configurations comparable accuracy. That gap is real and large.

The strengths: crossed-conflict design with verified evidence pairs, explicit trace filtering, paired bootstrap and McNemar tests, component analyses separating input order from instruction and provenance, and validation across gpt-oss, Gemma4, Qwen3 (plus DeepSeek-V4 for extra coverage). The trace-content controls (final line, extractive summary, unrelated trace) and the verification vs. renewed-reasoning comparison round it out. The paper earns its central claim.

Soft spots: the conflicts are constructed by editing gold evidence and retaining only pairs where both traces explicitly state their own answer. That is necessary for measurement but limits generalization to subtler real-world evidence changes or less explicit traces. The authors acknowledge this clearly. The Gemma4 continuation artifact (no-trace placeholder causing long outputs on 15% of MuSiQue) is minor and flagged. Residual leakage in evidence edits is discussed and cannot explain the dissociation because the gap far exceeds any plausible bias. The comparison across model families is solid but the main paired result rests on three model–dataset combinations; more would strengthen external validity.

Who this is for: researchers evaluating or building multi-step LLM systems, especially agent frameworks and RAG pipelines. The practical message—report both trace contribution and conflict response, plus the answer configuration—is actionable.

Recommendation: this is a well-executed study with a clear, non-obvious result. It deserves serious peer review and publication. I'd bring it to reading group and cite it in work on evidence-based answering.

The strongest objection

The paper's central claim—that trace use and conflict response dissociate—is well supported by controlled crossed-conflict experiments across three model families. The most plausible concerns (generalizability to less explicit traces and naturally updated evidence, the no-trace placeholder artifact for Gemma4-E4B continuation, and residual leakage in evidence edits) are either acknowledged by the authors as limitations or do not undermine the qualitative dissociation, because the gap in conflict response (29.9–60.6 points) far exceeds any plausible bias in the trace contribution measurement (≤4.7 points reported, and the acknowledged artifact affects only one model-dataset cell). The paper's experimental design is sound, and the dissociation is robust across models, datasets, and control analyses.

The paper presents a well-designed empirical study that cleanly dissociates trace use from conflict response. The central finding is robust across three model families, two datasets, and multiple control analyses (input-order variants, trace-content ablations, component analyses). The reader's weakest assumption—generalizability to naturally updated evidence, other tasks, and less explicit traces—is explicitly stated as a limitation by the authors. I find no internal inconsistency or unacknowledged assumption that could invalidate the core claim. The most concrete methodological concern, the no-trace placeholder artifact for Gemma4-E4B continuation, is acknowledged and affects only one model-dataset cell; even if corrected, the qualitative dissociation would persist. Hence, I do not recommend a change to the reader's acceptance verdict.

The referee's report

If the result holds, it provides a clear and actionable insight for the evaluation and design of multi-stage LLM systems: accuracy gains from a trace do not imply that the system will follow that trace when new evidence conflicts, and the answer configuration (e.g., separate call vs. continuation) is a major determinant of conflict response. The experimental methodology—crossed verified evidence pairs, model-specific trace filtering, paired bootstrap intervals and McNemar tests, and component analyses—is rigorous and sets a strong standard for future work. The dissociation is demonstrated across three model families and two datasets, with control analyses addressing confounds such as input order, trace provenance labels, and answer length.

Pith's reading of the paper (a machine's summary, not the authors')

This paper disentangles two roles of a reasoning trace retained at answer time: it can improve accuracy, and it can compete with updated evidence when they conflict. The authors construct controlled conflicts by pairing edited evidence versions with traces originally generated from the other version, then compare how models answer under continuation (appending the trace to the model's own thinking) versus a separate answer call (providing the trace as user text). Across three model families and two multi-hop QA datasets, the separate call follows the current evidence 29.9 to 60.6 points more often than continuation, even though removing the trace lowers accuracy by comparable amounts in both configurations. The paper concludes that benefitting from a trace and deferring to it are distinct properties, and that trace-deletion tests alone cannot detect which source an answer follows.

What carries the argument. A crossed-conflict experimental design that pairs two versions of the supporting evidence (original E0 and edited E1, with different target answers) with traces generated from each version, producing four cells per evidence pair: matched cells (Ei, ri) where trace and evidence agree, and crossed cells (Ei, rj) where they disagree. Every model receives identical traces and evidence under both continuation and separate answer call, so the answer configuration is the sole varying factor. Trace contribution is measured by comparing matched-cell accuracy with and without the trace; conflict response is measured by whether the crossed-cell answer matches the evidence target or the trace target.

Core claim. A reasoning trace retained at answer time serves two separable functions: it can raise answer accuracy (trace use), and it can steer the answer when the trace and the current evidence disagree (conflict response). These two properties dissociate sharply depending on how the answer is configured. In continuation, where the trace sits inside the model's own thinking block and the answer is generated directly after it, the model repeats the stale conclusion of the trace in 65.6–87.0% of conflicts. In a separate answer call, where the trace is supplied as user text and the evidence appears last, the model follows the current evidence 29.9–60.6 points more often on identical trace–evidence pairs. Yet removing the trace from the same pairs costs both configurations comparable accuracy (at most 4.7 points difference). The dissociation holds across gpt-oss-20b, Gemma4-E4B, and Qwen3-14B on HotpotQA and MuSiQue, and input order accounts for most of the gap in two of the three models.

The premise it rests on. The dissociation between trace use and conflict response may depend on the controlled, explicit nature of the evidence edits and the trace filtering; in real-world scenarios where evidence changes more subtly or traces are less explicit, the answer configuration might have a smaller or different effect.

If the paper is right (Pith's inferences, not the authors')

Evaluations of evidence-based LLM answers should report both trace contribution and conflict response together with the answer configuration (trace position, whether thinking is on, input order, instruction, output budget), because trace-deletion results do not reveal which source an answer follows.

For systems that receive new evidence after reasoning, a separate answer call is the cheaper correction and renewed reasoning (a new reasoning call over the current evidence) is the stronger one; a verification instruction alone does not loosen the hold of a stale trace.

Evidence quality limits correction: distractors weaken the evidence-following of a separate call, and evidence helps most when it arrives before the trace is written, so retrieval improvements should target the reasoning stage rather than the answer stage when possible.

The final line of a trace carries most of its value, making the answer stage vulnerable to copying a wrong conclusion from a stale trace; this motivates methods that separate answer generation from the reasoning trace more cleanly than continuation.

The dissociation implies that the mechanism by which a trace influences the answer is not a single 'reliance' strength but at least two separate pathways: a content pathway (the trace provides facts that the answer draws on) and an anchoring pathway (the trace locks the answer into a prior reasoning path). The answer configuration appears to gate the anchoring pathway more strongly than the conten

If input order accounts for most of the gap in two models, that suggests recency bias (the last input seen dominates) is a major driver of conflict resolution. This connects to the 'lost in the middle' literature and implies that simply placing the evidence after the trace in a separate call could substantially increase evidence-following, even without changing the model's training.

The finding that an instruction to verify the trace does not reduce trace-following (73.6% vs 72.2% for continuation) suggests that instruction-level interventions may be insufficient; structural changes to the answer procedure (e.g., separate call, renewed reasoning) are more reliable ways to override stale traces.

The asymmetry between the two conflict directions (trace-following is higher when the stale trace states the original answer, which matches parametric knowledge) implies that evaluation of conflict response should report directions separately, because a model's own knowledge can masquerade as trace-following when the trace matches what the model already 'knows.'

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

A reasoning trace retained at answer time plays two roles: it can raise accuracy, and it can compete with the current evidence when the two disagree. We disentangle these roles by crossing verified evidence versions with the traces generated from each and answering from the same traces and evidence under two configurations: continuation of the model's own thinking and a separate answer call. Across three model families (gpt-oss, Gemma4, Qwen3) on HotpotQA and MuSiQue, the separate call raises evidence-following over continuation by 29.9 to 60.6 points. On the same pairs, removing the trace lowers accuracy by similar amounts under both configurations. Yet continuation repeats the stale conclusion in most conflicts, and the separate call follows the current evidence far more often. Benefiting from a trace and deferring to it are therefore distinct properties. Further analyses show that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning, unlike a verification instruction, follows the current evidence in most conflicts. Because trace deletion alone cannot reveal which source an answer follows, evaluations should report both properties together with the answer configuration.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.82

“Benefiting from a trace and deferring to it are therefore distinct properties.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.69 · human participants 0.03 · controlled comparison 0.83 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.70 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
A reasoning trace retained at answer time plays two roles: it can raise accuracy, and it can compete with the current evidence when the two disagree. background 1.00 0.14 0.10 asserted 0.75
We disentangle these roles by crossing verified evidence versions with the traces generated from each and answering from the same traces and evidence under two configurations: continuation of the model's own thinking and a separate answer call. method 0.92 0.94 0.05 asserted 0.93
Across three model families (gpt-oss, Gemma4, Qwen3) on HotpotQA and MuSiQue, the separate call raises evidence-following over continuation by 29.9 to 60.6 points. result 1.00 0.96 0.31 asserted 1.00
On the same pairs, removing the trace lowers accuracy by similar amounts under both configurations. result 1.00 0.95 0.08 asserted 1.00
Yet continuation repeats the stale conclusion in most conflicts, and the separate call follows the current evidence far more often. result 0.99 0.93 0.28 asserted 1.00
Benefiting from a trace and deferring to it are therefore distinct properties. interpretation 0.98 0.79 0.59 asserted 1.00
Further analyses show that the final line of a trace carries most of its value, gold evidence helps most before the trace is written, distractors weaken correction under a separate call, and renewed reasoning, unlike a verification instruction, follows the current evidence in most conflicts. result 0.99 0.93 0.10 asserted 1.00
Because trace deletion alone cannot reveal which source an answer follows, evaluations should report both properties together with the answer configuration. future work 0.86 0.42 0.54 asserted 0.99

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 6,289 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (4)
Figure 1 of the paper
Figure 1. A crossed conflict on a HotpotQA question, with the outputs of Gemma4-E4B (evidence and trace
Figure 2 of the paper
Figure 2. The two answer configurations on identical crossed-conflict pairs, traces, and gold evidence. Each bar
Figure 3 of the paper
Figure 3. Stage-separated ablations. (a) Share of answers that change after removing the trace or all evidence at
Figure 4 of the paper
Figure 4. Repair of distractor-written failures. Circles give the repair rate when gold evidence replaces the distractors

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment