{"id":"729b4a29-d3e6-4da3-9bd3-4bc42b8007e7","arxiv_id":"2602.11619","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Behavioral inconsistency across repeated runs of an LLM agent correlates with lower task accuracy, but the paper's main abstract claims are not supported by its body.","lead":"Run the same AI agent twice on the same task and it often takes different action paths; this paper claims that inconsistent paths flag likely errors. A plausible idea, but the submitted text's abstract promises validation results (SWE-bench, AUROC, calibration) that never appear in the body.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Consistency metric is undefined: central correlation switches between tool-type and query-sensitive action definitions.","rationale":"The reader's weakest assumption correctly identifies the metric-identity ambiguity as the most load-bearing flaw. My independent reading confirms that §3.3 defines action sequences without arguments, while §4.3 treats the first search query as a point of divergence, which requires query text to be part of the action. These two definitions produce fundamentally different diversity scores, and the paper's numbers (2.0–4.2 unique sequences per 10 runs, 25–29 tasks in the extreme bins) are only consistent with the tool-type definition, yet the 'early divergence' story only makes sense with the query-sensitive definition. This ambiguity directly undermines the headline correlation. I also note the abstract/body discrepancies the reader mentions (8,000 vs. 3,000 runs, AUROC absent from body), but I treat the metric issue as more central because it invalidates the primary result even within the body's own scope. The reader's verdict of REJECT is appropriate; I do not see a way to salvage the central claim without a clear metric and re-analysis. The proposed test would settle the concern by forcing a choice of definition and checking robustness.","tokens_in":5607,"tokens_out":3306,"duration_ms":35820,"concrete_test":"Re-run the agent on the same 100 HotpotQA tasks (or a 200-task subset) with 10 runs per task, and compute Table 2 twice: (A) action = tool type only, as in §3.3; (B) action = tool type plus the full argument string (e.g., the actual Search query). Compare the unique-sequence distributions and the consistency–accuracy gaps for the ≤2 and ≥6 bins. If definition (B) yields fewer than 5% of tasks in the ≤2 bin or the gap falls below 10pp, the central correlation is an artifact of the coarse metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that behavioral consistency predicts correctness hinges on 'Action Sequence Diversity' (§3.3), defined as unique tool-type sequences like Search→Retrieve→Finish. But §4.3 measures divergence at 'step 2, the first search query', implying different query strings count as different actions. These definitions are incompatible. If the metric is tool types only, then runs that search for entirely different documents are coded as 'consistent', which would dilute—or even reverse—the consistency–accuracy gap. If the metric includes query text, then 10 runs of a stochastic LLM agent would rarely produce only 2.0–4.2 unique sequences; the reported distributions in Table 1 and the 25–29 task counts in Table 2 become implausible. The paper never states which definition generates Table 2, and the two analyses (correlation vs. early-divergence) rely on contradictory operationalizations. This is not a minor ambiguity: the entire selective prediction argument rests on the claim that 'consistent' tasks are genuinely similar in behavior, not merely similar in tool type. Without a fixed metric, the 32–55pp gap cannot be interpreted as evidence for the paper's conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies whether repeated runs of the same LLM agent on the same HotpotQA task produce similar action sequences, and whether such behavioral consistency correlates with correctness. Using a ReAct-style agent with Search, Retrieve, and Finish tools, the body reports 3,000 runs (100 tasks × 10 runs × 3 models) and finds 2.0–4.2 unique action sequences per task on average, with a 32–55 percentage-point accuracy gap between tasks having ≤2 unique sequences and those having ≥6. It further reports that divergence concentrates at step 2, that path length correlates negatively with correctness, and that temperature 0.0 improves consistency and accuracy. The abstract supplied for review, however, describes a substantially larger study with 8,000 runs, four models, 200 HotpotQA questions, AUROC values, selective prediction, split-conformal calibration, SWE-bench validation, and bootstrap misranking estimates; none of these experiments or analyses appear in the body.","tokens_in":5832,"tokens_out":8124,"duration_ms":86736,"significance":"If the consistency–correctness correlation were established, a training-free, black-box uncertainty signal for LLM agents would be practically valuable for monitoring and selective deployment. The manuscript's strengths are its use of an external benchmark, its lack of fitted parameters, and the promised code/data release. However, the submitted version does not establish the headline claims: the abstract and body describe different experiments, the central action-sequence metric is underspecified, the statistical support is thin, and the temperature ablation is not controlled. The credible core is the descriptive observation that run-to-run behavioral variance exists and tracks accuracy, but the current evidence is insufficient for the strong claims made.","major_comments":[{"comment":"The abstract claims 8,000 runs of four models on 200 HotpotQA questions, AUROC 0.62–0.78, selective prediction with k=3 agreement, a split-conformal baseline, SWE-bench validation, and bootstrap model-misranking estimates. Sections 3.2 and 4 describe 100 tasks × 10 runs × 3 models = 3,000 runs and none of those analyses. The abstract's statement that behavioral consistency 'instantiates selective classification and distribution-free calibration' is therefore unsupported by any experiment in the manuscript. This is a load-bearing discrepancy: either the body must be expanded to include those experiments, or the abstract must be rewritten to describe only what is actually shown.","section":"Abstract vs. §§3–4"},{"comment":"Action Sequence Diversity is defined in §3.3 as the number of unique action sequences 'e.g., Search→Retrieve→Finish', i.e. sequences of tool-call types with no mention of arguments. But §4.3 measures divergence at 'step 2, the first search query', implying that different query strings count as different actions. These are incompatible operationalizations. If the metric is tool types only, runs that search for entirely different documents are coded as consistent, which would dilute the consistency–accuracy gap; if the metric includes query text, the reported 2.0–4.2 unique sequences per 10 runs in Table 1 and the small inconsistent-group counts in Table 2 become implausible. The paper must state exactly what an 'action' is, how query text is compared, and report the main results under both granularities.","section":"§3.3 vs. §4.3"},{"comment":"The temperature ablation is not controlled for question set. The 0.0 row is based on a subset of 20 questions, while the 0.7 row is the full 100-question result from Table 1. The +5.4pp accuracy difference and the 4.2→2.2 unique-sequence difference could therefore reflect question-selection effects rather than temperature. To support the 'Temperature matters' contribution, both temperatures must be run on the same question set, or the temperature claim should be removed.","section":"§4.5, Table 4"},{"comment":"The consistency groups use thresholds of ≤2 and ≥6 unique sequences that are not justified, and no confidence intervals or error bars are reported. For Claude and GPT-4o the inconsistent bin has only n=9 and n=10 tasks, respectively, so the 41.5pp and 55.1pp gaps are estimated from very small samples. A significance test is reported only for Llama, and the binning is chosen after seeing the data. Moreover, consistency and correctness are computed on the same 10 runs, so the result is an in-sample descriptive correlation, not demonstrated predictive validity. Report a threshold sweep with standard errors or an ordinal analysis, and separate the descriptive correlation from the claim that consistency 'predicts' correctness.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The denominator for the 69% divergence figure is not stated. The table shows 59 of 86 tasks = 68.6%, and it is unclear how tasks that never diverge are treated. Please state the denominator and clarify.","section":"§4.3, Table 3"},{"comment":"The abstract in the body differs from the abstract supplied for review in run counts, model counts, and reported analyses. The manuscript also has inconsistent titles. Ensure the final version is internally consistent.","section":"Abstract/Title"},{"comment":"Figure 1 (a and b) is referenced in §4.1 and §4.2 but not included in the text. The figures are needed to evaluate the distributional claims.","section":"Figures"},{"comment":"Reference entries are inconsistently formatted (e.g., some have only titles and years, with missing venues or page information). Please harmonize.","section":"References"},{"comment":"The Limitations section acknowledges the small temperature-ablation sample but not that the 0.0 and 0.7 conditions use different question sets. This should be corrected.","section":"Limitations"}],"recommendation":"reject","confidential_remarks":"As submitted, the abstract and the body describe substantially different experiments, and the central consistency metric is ambiguous between tool-type sequences and query-sensitive actions. These are not local presentation issues: the paper's headline results cannot be evaluated or reproduced without resolving them. If the abstract's experiments exist in another version, the authors should resubmit that version; as it stands, the manuscript is not internally coherent enough for review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a small empirical note with one genuinely interesting measurement — on a ReAct agent over HotpotQA, repeated runs tend to diverge at the first search query, and tasks where runs agree are substantially more accurate. That observation is new relative to τ-bench and self-consistency work. But the paper currently cannot support its own abstract, and the central metric is ambiguous enough that the headline gap might be an artifact.\n\nWhat it does well: 3,000 runs across three models on an external benchmark, with code and data linked. The step-2 divergence claim is concrete and checkable. The consistency-accuracy gap, if real, is practically useful. The related work is honestly positioned.\n\nThe problems are not minor. The abstract reports 8,000 runs, four models, 200 questions, AUROC, SWE-bench, and split-conformal calibration; the body reports 3,000 runs, three models, 100 questions, and none of the AUROC/calibration/SWE-bench analyses. That is not a difference in emphasis. A reader cannot tell which study is being described. Second, 'action sequence diversity' is defined in §3.3 as unique tool-type tuples (Search→Retrieve→Finish), but §4.3 measures divergence at 'the first search query,' which only makes sense if query text is part of the action. These two operationalizations are incompatible. If the metric is tool types only, runs that search for entirely different content are coded as consistent, which could manufacture the consistency-accuracy gap. If it includes query text, 10 runs of a stochastic agent would almost never produce only 2.0–4.2 unique sequences. The paper never states which definition generates Table 2. Third, the consistent/inconsistent split (≤2 vs ≥6) is arbitrary, two of the inconsistent groups have n=9 and n=10, and significance is reported only for Llama. The temperature ablation compares 20 questions against 100. The claim that the gap survives controls for task difficulty appears only in the abstract; the body contains no such control.\n\nWho this is for: people building agent monitoring or selective-deployment systems. The core idea is worth pursuing, and with a clean metric definition, error bars, and an abstract that matches the body, this could be a solid workshop or short-conference paper. I would send it to peer review with a strong request to fix those issues; I would not cite it in its current form.","headline":"The body has a plausible small empirical study, but the abstract reports a different, much larger study, and the central consistency metric is ambiguous enough that the headline gap could be an artifact.","tokens_in":6321,"tokens_out":5694,"would_cite":false,"duration_ms":63911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Behavioral consistency of an LLM agent across repeated runs predicts answer correctness, providing a training-free, black-box uncertainty signal.","keywords":["LLM agents","behavioral consistency","uncertainty estimation","selective prediction","self-consistency","action sequences","agent reliability","calibration"],"falsifier":"Run the same agent 10 times on a set of question-answering tasks and compute consistency two ways: once as the number of distinct tool-call type sequences, and once as the number of distinct full action sequences including query strings. If the accuracy gap between 'consistent' and 'inconsistent' tasks shrinks or disappears under the query-text definition, then the paper's central signal is an artifact of a too-coarse metric.","tokens_in":5461,"feed_emoji":"🤖","tokens_out":5724,"duration_ms":51827,"temperature":0.7,"pith_summary":"This paper tries to establish that the variability in an LLM agent's behavior across repeated runs of the same task is a reliable, training-free signal of whether the agent's answer is correct. When an agent follows the same action sequence every time, it is right 80–92% of the time; when it produces many distinct sequences, accuracy drops to 25–60%. The paper interprets this behavioral variance as a black-box uncertainty signal that can drive selective prediction and calibration without ground-truth labels. If correct, this gives practitioners a low-cost way to flag unreliable agent outputs and decide when to intervene.","feed_headline":"Agent consistency predicts whether its answer is correct","feed_subtitle":"Repeated runs are a free uncertainty signal: if the agent keeps diverging, expect a wrong answer.","key_machinery":"The central object is 'action sequence diversity' — the number of distinct sequences of tool calls (e.g., Search→Retrieve→Finish) an agent generates when run repeatedly on an identical input. This metric requires no labels or training and is computed from the agent's own trajectory. The paper uses it as a proxy for the agent's uncertainty: high diversity indicates the agent is 'searching, backtracking, and uncertain,' and the analysis shows that divergence at the first search query (step 2) is the dominant source of variance (69% of tasks).","core_discovery":"The central claim is that action-sequence consistency across repeated runs predicts answer correctness. Defining consistency as the number of unique sequences of tool calls an agent produces over 10 runs, the authors report a large and statistically significant gap: tasks with one or two unique sequences achieve 80–92% accuracy, while tasks with six or more achieve only 25–60%. They attribute this to early decision divergence, primarily at the first search query, which sets the trajectory for the rest of the run. The authors argue that this consistency signal is 'training-free, black-box' and enables selective classification and distribution-free calibration for agentic systems.","pith_inferences":["If the consistency signal proves robust to the metric ambiguity (whether 'different actions' counts query text), it could generalize to any stochastic agent, giving a universal, label-free uncertainty estimator.","The concentration of divergence at the first search query suggests a targeted fix: improving query generation, e.g., with query expansion or deterministic search, may reduce downstream variance and boost reliability.","A direct test: compare consistency's predictive power against model confidence scores (e.g., token logprobs) on the same tasks to see whether it captures complementary uncertainty information.","The discrepancy between the abstract's larger-scale results (8,000 runs, four models, SWE-bench) and the full text's 3,000 runs and three models is unresolved; reproducing the selective-prediction and calibration claims at the reported scale is the next check."],"forward_implications":["Runtime monitoring of consistency can trigger early error detection or human review before an agent commits to an answer.","Selective prediction — answering only when several runs agree — can lift accuracy by 6–14 percentage points over single-run baselines, without requiring a held-out calibration set.","Consistency can serve as a model-selection criterion: more reliable agents show less behavioral variance.","Lowering sampling temperature is a simple lever that reduces both variance and error, offering a practical deployment tweak.","Longer, more variable trajectories are more error-prone, suggesting that agents that backtrack should be treated as less trustworthy."],"fun_headline_variants":["Agent inconsistency predicts answer accuracy","When agents disagree with themselves: a reliability signal","Repeated runs: a training-free uncertainty measure for agents","Consistency gap: 82-87% vs 41-65% accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's consistency metric is defined two, incompatible ways — as tool-call types and as the text of the first search query — and the consistency–accuracy gap changes depending on which definition is used; the paper never resolves this.","fun_headline_variants_meta":{"raw":{"variants":["Agent inconsistency predicts answer accuracy","When agents disagree with themselves: a reliability signal","Repeated runs: a training-free uncertainty measure for agents","Consistency gap: 82-87% vs 41-65% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1319,"prompt_tokens":749,"completion_tokens":570,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":506}},"tokens_in":493,"tokens_out":570,"duration_ms":6133,"temperature":1.0,"reasoning_tokens":506,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:03:48.165954+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same agent 10 times on a set of question-answering tasks and compute consistency two ways: once as the number of distinct tool-call type sequences, and once as the number of distinct full action sequences including query strings. If the accuracy gap between 'consistent' and 'inconsistent' tasks shrinks or disappears under the query-text definition, then the paper's central signal is an artifact of a too-coarse metric.","supporting_citations":[],"review_version":1}