REVIEW 4 major objections 5 minor 2 cited by
When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents
T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Behavioral consistency of an LLM agent across repeated runs predicts answer correctness, providing a training-free, black-box uncertainty signal.
desk verdict The body has a plausible small empirical study, but the abstract reports a different, much larger study, and the central consistency metric is ambiguous enough that the headline gap could be an artifact. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is 'action sequence diversity' — the number of distinct sequences of tool calls (e.g., Search→Retrieve→Finish) an agent generates when run repeatedly on an identical input. This metric requires no labels or training and is computed from the agent's own trajectory. The paper uses it as a proxy for the agent's uncertainty: high diversity indicates the agent is 'searching, backtracking, and uncertain,' and the analysis shows that divergence at the first search query (step 2) is the dominant source of variance (69% of tasks).
What would settle it
Run the same agent 10 times on a set of question-answering tasks and compute consistency two ways: once as the number of distinct tool-call type sequences, and once as the number of distinct full action sequences including query strings. If the accuracy gap between 'consistent' and 'inconsistent' tasks shrinks or disappears under the query-text definition, then the paper's central signal is an artifact of a too-coarse metric.
Extended reading notes
Core claim
The central claim is that action-sequence consistency across repeated runs predicts answer correctness. Defining consistency as the number of unique sequences of tool calls an agent produces over 10 runs, the authors report a large and statistically significant gap: tasks with one or two unique sequences achieve 80–92% accuracy, while tasks with six or more achieve only 25–60%. They attribute this to early decision divergence, primarily at the first search query, which sets the trajectory for the rest of the run. The authors argue that this consistency signal is 'training-free, black-box' and enables selective classification and distribution-free calibration for agentic systems.
Load-bearing premise
The paper's consistency metric is defined two, incompatible ways — as tool-call types and as the text of the first search query — and the consistency–accuracy gap changes depending on which definition is used; the paper never resolves this.
Editorial extensions
If this is right
- Runtime monitoring of consistency can trigger early error detection or human review before an agent commits to an answer.
- Selective prediction — answering only when several runs agree — can lift accuracy by 6–14 percentage points over single-run baselines, without requiring a held-out calibration set.
- Consistency can serve as a model-selection criterion: more reliable agents show less behavioral variance.
- Lowering sampling temperature is a simple lever that reduces both variance and error, offering a practical deployment tweak.
- Longer, more variable trajectories are more error-prone, suggesting that agents that backtrack should be treated as less trustworthy.
Reading between the lines
- If the consistency signal proves robust to the metric ambiguity (whether 'different actions' counts query text), it could generalize to any stochastic agent, giving a universal, label-free uncertainty estimator.
- The concentration of divergence at the first search query suggests a targeted fix: improving query generation, e.g., with query expansion or deterministic search, may reduce downstream variance and boost reliability.
- A direct test: compare consistency's predictive power against model confidence scores (e.g., token logprobs) on the same tasks to see whether it captures complementary uncertainty information.
- The discrepancy between the abstract's larger-scale results (8,000 runs, four models, SWE-bench) and the full text's 3,000 runs and three models is unresolved; reproducing the selective-prediction and calibration claims at the reported scale is the next check.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript studies whether repeated runs of the same LLM agent on the same HotpotQA task produce similar action sequences, and whether such behavioral consistency correlates with correctness. Using a ReAct-style agent with Search, Retrieve, and Finish tools, the body reports 3,000 runs (100 tasks × 10 runs × 3 models) and finds 2.0–4.2 unique action sequences per task on average, with a 32–55 percentage-point accuracy gap between tasks having ≤2 unique sequences and those having ≥6. It further reports that divergence concentrates at step 2, that path length correlates negatively with correctness, and that temperature 0.0 improves consistency and accuracy. The abstract supplied for review, however, describes a substantially larger study with 8,000 runs, four models, 200 HotpotQA questions, AUROC values, selective prediction, split-conformal calibration, SWE-bench validation, and bootstrap misranking estimates; none of these experiments or analyses appear in the body.
Significance. If the consistency–correctness correlation were established, a training-free, black-box uncertainty signal for LLM agents would be practically valuable for monitoring and selective deployment. The manuscript's strengths are its use of an external benchmark, its lack of fitted parameters, and the promised code/data release. However, the submitted version does not establish the headline claims: the abstract and body describe different experiments, the central action-sequence metric is underspecified, the statistical support is thin, and the temperature ablation is not controlled. The credible core is the descriptive observation that run-to-run behavioral variance exists and tracks accuracy, but the current evidence is insufficient for the strong claims made.
major comments (4)
- [Abstract vs. §§3–4] The abstract claims 8,000 runs of four models on 200 HotpotQA questions, AUROC 0.62–0.78, selective prediction with k=3 agreement, a split-conformal baseline, SWE-bench validation, and bootstrap model-misranking estimates. Sections 3.2 and 4 describe 100 tasks × 10 runs × 3 models = 3,000 runs and none of those analyses. The abstract's statement that behavioral consistency 'instantiates selective classification and distribution-free calibration' is therefore unsupported by any experiment in the manuscript. This is a load-bearing discrepancy: either the body must be expanded to include those experiments, or the abstract must be rewritten to describe only what is actually shown.
- [§3.3 vs. §4.3] Action Sequence Diversity is defined in §3.3 as the number of unique action sequences 'e.g., Search→Retrieve→Finish', i.e. sequences of tool-call types with no mention of arguments. But §4.3 measures divergence at 'step 2, the first search query', implying that different query strings count as different actions. These are incompatible operationalizations. If the metric is tool types only, runs that search for entirely different documents are coded as consistent, which would dilute the consistency–accuracy gap; if the metric includes query text, the reported 2.0–4.2 unique sequences per 10 runs in Table 1 and the small inconsistent-group counts in Table 2 become implausible. The paper must state exactly what an 'action' is, how query text is compared, and report the main results under both granularities.
- [§4.5, Table 4] The temperature ablation is not controlled for question set. The 0.0 row is based on a subset of 20 questions, while the 0.7 row is the full 100-question result from Table 1. The +5.4pp accuracy difference and the 4.2→2.2 unique-sequence difference could therefore reflect question-selection effects rather than temperature. To support the 'Temperature matters' contribution, both temperatures must be run on the same question set, or the temperature claim should be removed.
- [§4.2, Table 2] The consistency groups use thresholds of ≤2 and ≥6 unique sequences that are not justified, and no confidence intervals or error bars are reported. For Claude and GPT-4o the inconsistent bin has only n=9 and n=10 tasks, respectively, so the 41.5pp and 55.1pp gaps are estimated from very small samples. A significance test is reported only for Llama, and the binning is chosen after seeing the data. Moreover, consistency and correctness are computed on the same 10 runs, so the result is an in-sample descriptive correlation, not demonstrated predictive validity. Report a threshold sweep with standard errors or an ordinal analysis, and separate the descriptive correlation from the claim that consistency 'predicts' correctness.
minor comments (5)
- [§4.3, Table 3] The denominator for the 69% divergence figure is not stated. The table shows 59 of 86 tasks = 68.6%, and it is unclear how tasks that never diverge are treated. Please state the denominator and clarify.
- [Abstract/Title] The abstract in the body differs from the abstract supplied for review in run counts, model counts, and reported analyses. The manuscript also has inconsistent titles. Ensure the final version is internally consistent.
- [Figures] Figure 1 (a and b) is referenced in §4.1 and §4.2 but not included in the text. The figures are needed to evaluate the distributional claims.
- [References] Reference entries are inconsistently formatted (e.g., some have only titles and years, with missing venues or page information). Please harmonize.
- [Limitations] The Limitations section acknowledges the small temperature-ablation sample but not that the 0.0 and 0.7 conditions use different question sets. This should be corrected.
Circularity Check
No circular derivation: the consistency–correctness correlation is an empirical measurement on an external benchmark, with no fitted parameters, no prediction derived from labels, and no load-bearing self-citation chain.
full rationale
The paper's central claim is that action-sequence diversity (§3.3) correlates with correctness (§3.3) across HotpotQA runs. These two quantities are measured independently: diversity counts unique action sequences, while correctness uses fuzzy string matching of the final answer against a gold answer. No equation defines one in terms of the other, and no parameter is fitted to accuracy labels and then renamed as a prediction. The consistency groups in Table 2 are descriptive strata of the same 10 runs used for accuracy, which raises a statistical in-sample concern, but it is not circularity by construction: the correlation could have failed to appear and is not guaranteed by the metric definitions. The paper also relies on external benchmarks (HotpotQA, and the abstract's SWE-bench validation) and external prior work (ReAct, τ-bench, self-consistency), with no self-citation chain supporting the main result. The ambiguity between tool-type action sequences (§3.3) and query-sensitive divergence at step 2 (§4.3) is a definitional inconsistency that threatens the validity of the measured correlation, but it does not make the correlation true by definition and is not a circular reduction. The paper explicitly acknowledges its limitations (single benchmark, lexical search, small temperature ablation), which further supports treating the findings as empirical rather than tautological. No circular step meeting the required evidence standard was found.
Assumptions & free parameters
free parameters (4)
- Consistency thresholds =
2 and 6 unique sequences
- Runs per task N =
10
- Temperature =
0.7 (main), 0.0 (ablation)
- Selective threshold k =
3
assumptions (5)
- domain assumption Action sequence diversity counts tool-call types, not arguments.
- domain assumption The 100 HotpotQA 'hard' questions are a representative sample.
- domain assumption Keyword matching over the HotpotQA context is a sufficient retrieval proxy.
- domain assumption Fuzzy string containment is an adequate correctness criterion.
- domain assumption The consistency-accuracy correlation measured in-sample on the same 10 runs transfers to selective prediction on new tasks.
Cite this review
Pith. "Pith review of When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents." pith.science (2026). https://pith.science/paper/PVQVSZVI
@misc{pith2026260211619,
author = {Pith},
title = {Pith review of: When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/PVQVSZVI}},
note = {Machine review of arXiv:2602.11619}
}
read the original abstract
Running the same LLM agent on identical inputs yields 2.3-4.2 distinct action sequences per 10 runs; this behavioral variance constitutes a training-free, black-box uncertainty signal that instantiates selective classification and distribution-free calibration for agentic systems. Across 8,000 runs of four models on 200 HotpotQA questions, consistent tasks (at most 2 unique paths) achieve 82-87% accuracy while inconsistent tasks (4 or more paths) achieve 41-65%, a gap that survives controls for task difficulty. Divergence concentrates at step 2 (50.5% of Llama tasks), and consistency metrics detect failures with AUROC 0.62-0.78. Exploiting this signal, selective prediction (answering only when k=3 runs agree) achieves 87-88% accuracy at 54-62% coverage, a 6-14pp gain over single-run baselines, and matches a split-conformal baseline without a held-out calibration set. A cross-benchmark validation on SWE-bench (50 tasks, 1,000 runs) preserves the consistency hierarchy while revealing an ~8x spread in mean trajectory length across models, and bootstrap analysis shows single-run evaluations misrank models 29.3% of the time.
Figures
Forward citations
Cited by 2 Pith papers
-
One Run Is Not an Idea: The Implementation Lottery in Automated Research
In audits of 312 LLM research assignments, variation across implementations of the same idea dominates rerun noise and reverses the selected idea winner in 25.6-43.6% of decisions.
-
Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis
Architecturally scaling LLM multi-agent systems raises accuracy at near-linear cost only above a model-capability threshold, peaks at intermediate complexity, and never fixes poor run-to-run consistency.
Reference graph
Works this paper leans on
-
[1]
Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. Language models (mostly) know what they know.arXiv preprint arXiv:2207.05221,
-
[3]
Autogen: Enabling next-gen llm applications via multi-agent conversation
Wu, Q., Bansal, G., Zhang, J., Wu, Y ., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155,
-
[4]
E., Wettig, A., Liber, K., Narasimhan, K., and Press, O
Yang, J., Jimenez, C. E., Wettig, A., Liber, K., Narasimhan, K., and Press, O. Swe-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793,
-
[5]
Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,
-
[2023]
Ai agents that matter.arXiv preprint arXiv:2407.01502,
Stroebl, B., Kapoor, S., and Narayanan, A. Ai agents that matter.arXiv preprint arXiv:2407.01502,
-
[2024]
F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y ., Fried, D., Alon, U., et al
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y ., Fried, D., Alon, U., et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854,
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.