{"id":"964f10b5-3059-44ad-87f1-a66edb805e5c","arxiv_id":"2608.09898","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selecting LLM rollouts with low early confidence and high final confidence outperforms confidence-maximizing methods on hard math and free-form code benchmarks.","lead":"This paper proposes a new way to choose the best answer when an AI model generates many candidate solutions: favor traces that start uncertain and end confident, instead of traces that are confident from the very beginning. This 'consilience' score improves test-time scaling on hard math and code problems, where standard confidence-based selection often fails.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing length control: S = C_final − α·C_initial may simply select longer exploratory traces on hard problems; without a longest-completion or length-matched baseline, the central temporal-asymmetry mechanism is underdetermined.","rationale":"Good-faith reading: the empirical work is careful—shared pools, repeated draws, 5-fold CV, cross-dataset transfer, and paired statistics. The -full ablation and reliable delimiters for GPT-OSS/DeepSeek give the practical score independent support. However, the metric's defining feature (penalizing early confidence) is never isolated from the natural correlation between long hard-problem traces and low early confidence; the paper's own Figure 6 case study shows the consilience pick is much longer than the mean-confidence pick. A missing length control is an omitted comparison that can overturn the mechanistic claim: it would show whether 'temporal asymmetry' or simply 'more tokens before converging' drives selection. This is more load-bearing than the reader's segmentation concern, which applies only to Qwen heuristics and is partly mitigated by Cns-full on the same datasets. If the length control fails, the central consilience mechanism is not supported; if it passes, the paper's interpretation is substantially strengthened. Hence CONDITIONAL rather than outright ACCEPT or REJECT.","tokens_in":25699,"tokens_out":18926,"duration_ms":187565,"concrete_test":"On the same completion pools as Table 2/Figure 1, compute the Top-1 accuracy of a 'longest completion' baseline (ties broken by C_final) and compare it with consilience; then residualize S on log sequence length and recompute the AUROC on the hard (<20% Pass@1) subsets. If the longest-completion baseline matches the reported gains, or if length-residualized S drops to chance AUROC, the temporal-asymmetry interpretation fails. If S remains predictive within length quartiles, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that the temporal asymmetry of confidence—low initial, high final—selects correct rollouts. But in Eqs. (3)–(4), both windows are entangled with sequence length: C_initial averages the first 20% of tokens after a 5% skip, so a long trace's initial window covers more early exploration and is naturally low-confidence, while a short trace's same relative window may already be mid-solution. On hard problems (Pass@1<20%), correct completions are likely longer because solving them requires more steps. Consequently S may reduce to 'prefer longer, more exploratory-looking outputs.' The paper reports no baseline selecting the longest completion, no AUROC for length, and no length-stratified analysis; Figure 3 and Table 4 therefore do not separate the consilience mechanism from a length proxy. This affects all models and datasets, unlike the Appendix B segmentation heuristics, which are model-specific and partially controlled by the -full ablation. If length explains the gains, the 'exploratory branching then convergence' interpretation is unsupported, even though the score might survive as an ad hoc heuristic.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes consilience, a verifier-free test-time scaling metric that selects LLM rollouts by their temporal confidence asymmetry: low initial confidence (interpreted as exploratory branching) resolving into high final confidence (interpreted as convergence). The score S = C_final − α·C_initial is computed from token-level top-K log-probabilities over initial and final windows, optionally restricted to the isolated reasoning phase. Experiments on LiveCodeBench, SWE-bench, HMMT, and GPQA across several models show that consilience outperforms mean-confidence baselines, especially on hard problems, and the paper includes robustness analyses for window sizes, α, and hyperparameter transfer.","tokens_in":26120,"tokens_out":6124,"duration_ms":56634,"significance":"If the result holds, the paper makes a useful contribution: it identifies a failure mode of confidence maximization (uniformly high confidence on hard problems), proposes a simple, training-free, logits-only selection metric that applies to free-form generation where majority voting cannot be used, and supports it with careful evaluation (shared pools, repeated subsampling, cross-validation, hyperparameter freezing, transfer, difficulty stratification, and statistical tests on the trajectory signal). The main unresolved risk is whether the selection gains reflect the temporal asymmetry itself or a length proxy, which the current analysis does not fully rule out.","major_comments":[{"comment":"The consilience score S = C_final − α·C_initial uses windows that scale with sequence length for the 20% variant and are fixed for the 2048-token variant. Because hard problems' correct completions are likely longer, S may systematically prefer longer, more exploratory traces, and the reported gains over mean-confidence baselines are not separated from a length proxy. The paper reports no baseline that selects the longest completion, no AUROC for completion length as a correctness signal, and no length-stratified selection analysis. Please add (i) a longest-completion baseline, (ii) length AUROC on hard problems, and (iii) selection results stratified by completion-length quartiles; if the gains persist after length matching, the temporal-asymmetry interpretation is supported.","section":"Section 3.4, Eq. (4), Tables 2–3"}],"minor_comments":[{"comment":"The sentence 'We calculate the consilience score from Equation 2' should reference Eq. (4), not Eq. (2).","section":"Section 4.2"},{"comment":"The word 'consiliencee' in 'with consiliencee can be efficient and effective' is a typo.","section":"Section 4.3"},{"comment":"The phrase 'the model generate large trunk of justification texts and reiterates on the answer' has grammatical errors; also the spelling 'interger' should be 'integer'.","section":"Appendix B"},{"comment":"The word 'evalutaion' in 'We use the evalutaion script supplied with the benchmark' should be 'evaluation'.","section":"Appendix A"},{"comment":"The hard-problem thresholds (Pass@1 <20% for the AUROC analysis and <50% for Figure 3) appear to be chosen post hoc; please state whether these thresholds were pre-specified and report AUROC across the full difficulty range to show that the signal is monotonic in difficulty.","section":"Section 4.4"},{"comment":"No paired significance tests are reported for the selection accuracy improvements; given that some confidence intervals overlap (e.g., GPT-OSS-20B/LCB, Pass@1 80.2±0.1 vs. Cns-think 81.1±0.7), please report paired tests across the repeated subsamples or clarify the aggregate-level interpretation.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the length confound; if the authors can rule it out with the suggested baselines, the paper is acceptable. The evaluation is otherwise solid. The post-hoc threshold and Qwen segmentation heuristics are secondary concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nYou should know that this is a serious empirical paper that introduces a simple, cheap selection metric for verifier-free test-time scaling. The central claim—rewarding low initial confidence and high final confidence in a rollout—holds up reasonably well in the experiments, but the paper never rules out the possibility that the score is largely a length proxy. If I were refereeing, I'd ask for a longest-completion baseline and a length-stratified analysis before accepting the mechanism story as stated.\n\nWhat's genuinely new: the active selection objective S = C_final − α·C_initial, the reasoning-phase isolation trick, and the application to free-form code generation and agentic steps where voting is impossible. The evaluation is careful: shared pools, repeated subsampling, hyperparameter transfer, difficulty stratification, AUROC and paired tests. The gains are consistent, though modest (a few points over Pass@1 and the confidence baselines). The paper also cites the concurrent work on premature confidence and confidence gain, and it reconciles the apparently conflicting result that maximizing prefix confidence helps on easy problems but hurts on hard ones. That is a fair reading of the literature.\n\nThe soft spots: the length confound is the one that bothers me most. Because C_initial is computed over the first W tokens (or relative percentage), a longer trace's initial window covers more genuine early exploration, and on hard problems correct completions are likely longer. The paper offers no baseline that selects the longest completion, no AUROC for length, and no length-stratified results. Figure 3 resamples trajectories to a common length, which shows the temporal pattern, but it doesn't isolate the effect from absolute length. The post-hoc 'hard problem' thresholds (<20% and <50%) are also a bit convenient, though the difficulty tier analysis in Table 4 mitigates that. The reasoning-phase isolation for non-thinking models relies on dataset-specific delimiters, but the -full ablation shows the score still works without that step, so this is a minor concern, not a fatal one.\n\nBottom line: the empirical result is probably real, and the score is cheap enough to be useful even if the 'consilience' interpretation is not fully established. The paper deserves a serious referee. I'd recommend accepting it with a request for a length-controlled experiment, or at least a discussion of the confound. I would bring it to the reading group and cite it in my own work if I were working on verifier-free TTS.","headline":"A careful empirical paper on a cheap selection metric for verifier-free test-time scaling; the central temporal-asymmetry mechanism still needs a length-control experiment.","tokens_in":26436,"tokens_out":3829,"would_cite":true,"duration_ms":33735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low early confidence predicts correct LLM reasoning","keywords":["test-time scaling","verifier-free selection","consilience","confidence trajectory","best-of-n sampling","large language models","reasoning","code generation"],"falsifier":"Compare consilience against its inverted form, $S' = C_{\\mathrm{initial}} - \\alpha\\, C_{\\mathrm{final}}$, on the same hard LiveCodeBench split: if the inverted score selects correct completions at least as often, or if the AUROC of $S$ versus correctness drops to 0.5 on a dataset where the reasoning delimiter is removed or mis-split, the asymmetry claim is not the operative mechanism.","tokens_in":25526,"feed_emoji":"📈","tokens_out":4980,"duration_ms":43165,"temperature":0.7,"pith_summary":"The paper argues that the standard way of doing verifier-free test-time scaling — sampling many completions and picking the one with the highest mean token confidence — systematically fails on hard problems, because uniformly high confidence often signals that the model committed to a single flawed path instead of exploring. It claims that correct reasoning traces have a characteristic temporal shape: low confidence early, when the model is still branching across candidate approaches, resolving into high confidence at the end, when the path converges on an answer. The authors propose consilience, a training-free selection score that subtracts a penalty for early confidence from the final confidence, and show that it beats confidence-maximization baselines and Pass@1 on graduate-level math, science, and free-form code generation, including agentic software editing. If the claim holds, verifier-free test-time scaling can be made reliable on exactly the open-ended tasks where external verifiers and majority voting do not apply.","feed_headline":"Low early confidence predicts correct LLM reasoning","feed_subtitle":"A score that rewards rising confidence beats mean-confidence selection and Pass@1 on hard math and code.","key_machinery":"The central object is the consilience score, a combinatorial metric $S = C_{\\mathrm{final}} - \\alpha\\, C_{\\mathrm{initial}}$ built from token-level confidence values (negative mean log-probability over the top-$K$ candidate tokens). It measures the temporal asymmetry of confidence across a reasoning trace, penalizing premature convergence (high initial confidence) while rewarding final certainty; a reasoning-phase isolation step strips away answer-summarization tokens so the metric reads only the cognitive search phase.","core_discovery":"On hard problems (where a model's Pass@1 is low), the distribution of mean token confidence inverts: incorrect completions are not only more confident on average but form a heavy tail of 'confidently wrong' answers, so maximizing mean confidence is worse than random selection. The paper's central discovery is that the temporal order of confidence carries the signal that the overall level does not: correct completions show a rising trajectory, with lower initial confidence (reflecting genuine exploration of multiple paths) and higher final confidence (reflecting convergence), whereas incorrect ones often commit early with high confidence and stay overconfident. The paper operationalizes this as the consilience score $S = C_{\\mathrm{final}} - \\alpha\\, C_{\\mathrm{initial}}$, computed from token-level top-$K$ log-probabilities, optionally restricted to the isolated reasoning phase of a thinking model, and demonstrates that selecting the completion with the highest $S$ improves accuracy across HMMT, GPQA, LiveCodeBench, and SWE-bench, with the largest gains on free-form code generation where voting is impossible.","pith_inferences":["The fixed penalty $\\alpha$ and window size could be replaced by a learned or calibrated function of task difficulty; the difficulty-stratified results suggest $\\alpha$ might ideally scale with problem hardness.","The same temporal-asymmetry signal could serve as an intrinsic reward for reinforcement learning, shaping models to explore early and converge late, beyond selection at inference.","For tree-search test-time scaling, consilience could be used to select trustworthy root trajectories or prune branches, mitigating the outsized influence of early samples.","The dependency on model-specific delimiters for reasoning isolation suggests a testable extension: prompting non-thinking models to emit a final-answer marker should make the method uniformly applicable."],"forward_implications":["Confidence-based test-time selection should evaluate the trajectory of confidence, not its average, on hard problems.","Verifier-free scaling can be applied to free-form code generation and agentic edits, where exact-match majority voting is impossible, with gains over Pass@1 (for example, GPT-OSS-120B on LiveCodeBench from 65.7% to 69.7%).","The metric is training-free, needs only token log-probabilities, and adds near-zero overhead, so it can be deployed through APIs that expose top-log-probs.","Penalizing high initial confidence is neutral on easy problems and beneficial on medium and hard tiers, explaining when prefix-confidence maximization helps and when it backfires.","Integrating consilience into an agentic loop improves SWE-bench resolution rates without full per-step scaling."],"supporting_citations":[{"why":"Supplies the Self-Certainty baseline that averages token confidence over the whole sequence for best-of-n selection.","marker":"[17]"},{"why":"Supplies the DeepConf baseline that computes confidence only over the final 2048 tokens, which consilience must beat.","marker":"[11]"},{"why":"Frames consilience as the principle underlying self-consistency voting, which consilience extends to free-form outputs.","marker":"[38]"},{"why":"Provides the apparently opposing finding that prefix-confidence maximization helps, which the paper reconciles by task difficulty.","marker":"[30]"},{"why":"Provides evidence that premature confidence predicts flawed reasoning and that gradual confidence growth is a valid training target.","marker":"[12]"},{"why":"LiveCodeBench is the free-form code generation benchmark where voting is inapplicable and consilience shows its largest gains.","marker":"[15]"},{"why":"SWE-bench supplies the agentic software engineering setting used to demonstrate consilience in multi-step editing.","marker":"[16]"},{"why":"GPQA supplies the graduate-level science benchmark used alongside HMMT for evaluation.","marker":"[31]"},{"why":"DeepSeek-R1 supplies one of the reasoning models evaluated, providing a dedicated end-of-think delimiter for reasoning isolation.","marker":"[14]"}],"fun_headline_variants":["Rising confidence beats mean confidence in LLM test-time scaling","Why high confidence often means wrong answer in LLM reasoning","Confidence trajectory, not level, predicts LLM solution correctness","Consilience score: penalize early confidence, reward final certainty","When LLMs are too confident early, they fail: new selection method"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the premise that on hard problems, tokens near the start of the reasoning trace measure genuine exploration through low confidence and tokens near the end measure convergence through high confidence — so the computed score reflects an actual cognitive trajectory, not artifacts of token position or segmentation.","fun_headline_variants_meta":{"raw":{"variants":["Rising confidence beats mean confidence in LLM test-time scaling","Why high confidence often means wrong answer in LLM reasoning","Confidence trajectory, not level, predicts LLM solution correctness","Consilience score: penalize early confidence, reward final certainty","When LLMs are too confident early, they fail: new selection method"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3215,"prompt_tokens":1048,"completion_tokens":2167,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":664,"completion_tokens_details":{"reasoning_tokens":2079}},"tokens_in":664,"tokens_out":2167,"duration_ms":13657,"temperature":1.0,"reasoning_tokens":2079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:41:56.178727+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare consilience against its inverted form, $S' = C_{\\mathrm{initial}} - \\alpha\\, C_{\\mathrm{final}}$, on the same hard LiveCodeBench split: if the inverted score selects correct completions at least as often, or if the AUROC of $S$ versus correctness drops to 0.5 on a dataset where the reasoning delimiter is removed or mis-split, the asymmetry claim is not the operative mechanism.","supporting_citations":[{"cited_title":"Maximizing Prefix-Confidence at Test-Time Efficiently Improves Mathematical Reasoning","cited_arxiv_id":"2507.18122","evidence_quote":"Provides the apparently opposing finding that prefix-confidence maximization helps, which the paper reconciles by task difficulty."},{"cited_title":"Zico Kolter, Andrej Risteski, and Aditi Raghunathan","cited_arxiv_id":null,"evidence_quote":"Provides evidence that premature confidence predicts flawed reasoning and that gradual confidence growth is a valid training target."}],"review_version":1}