{"id":"121b2c60-eb58-4be2-9e89-d5f518160c8d","arxiv_id":"2605.25244","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Correct reasoning traces exhibit positive confidence gain while incorrect traces show declining confidence, enabling CDG-based voting that boosts performance on AIME, HMMT and BRUMO benchmarks across multiple LLM architectures.","lead":"The paper observes that correct LLM reasoning traces show rising confidence over steps while incorrect ones show falling confidence, and proposes using this pattern in CDG voting to select better answers. A smart generalist might read it for a potential low-cost way to improve LLM reliability on hard problems like math contests.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Model confidence scores assumed to be reliable monotonic signal of trajectory correctness","rationale":"The load-bearing concern matches the reader's weakest assumption exactly. The UNVERDICTED verdict is appropriate given the abstract-only review; the same concern would persist even with full text unless the paper provides independent calibration checks or ablations on confidence aggregation.","tokens_in":1697,"tokens_out":279,"duration_ms":30863,"concrete_test":"On the collected reasoning traces, compute Pearson correlation between per-step confidence values and eventual answer correctness (binary label); if mean correlation across models/benchmarks is below 0.25 or if >30% of incorrect traces exhibit positive gain, the discriminative signal is too weak to support the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (correct traces show positive confidence gain; incorrect show decline) and the CDG voting method rest on the assumption that the LLM's reported token- or sequence-level confidence provides a reliable, monotonic indicator of whether the ongoing reasoning trajectory will reach a correct final answer. This assumption is least secure because LLMs are typically poorly calibrated on reasoning tasks, and confidence can be driven by superficial factors (token probabilities, formatting, or length) unrelated to correctness. If the signal is noisy or non-monotonic, the observed pattern and performance gains would not generalize.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript claims that correct LLM reasoning traces exhibit positive confidence gain over time while incorrect traces show attenuated or declining confidence; based on this pattern it introduces Confidence Dynamic Gain (CDG) voting that incorporates the evolution of confidence along the reasoning chain, reports significant gains over baselines on AIME24/25, HMMT25 and BRUMO25 across four models (DeepSeek-R1, gpt-oss, Gemma-3, Qwen-QwQ), and supplies theoretical insights.","tokens_in":1779,"tokens_out":367,"duration_ms":29080,"significance":"If the claimed pattern is robust and the CDG method generalizes beyond the reported benchmarks, the work would supply a lightweight, training-free inference-time signal for answer selection that exploits internal model uncertainty rather than external verification. The planned code release would strengthen reproducibility.","major_comments":[{"comment":"Abstract: the central empirical claim (positive vs. declining confidence trajectories) and the performance gains of CDG voting cannot be evaluated because the manuscript provides no description of how token- or sequence-level confidence is extracted, no exact voting formula, and no statistical tests or controls for multiple comparisons across models and benchmarks.","section":"Abstract"},{"comment":"Abstract: the load-bearing assumption that reported confidence scores supply a reliable monotonic signal of trajectory correctness is not supported by any calibration analysis, ablation on superficial factors (length, formatting), or comparison against known poor calibration of LLMs on reasoning tasks.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: 'for first time' should read 'for the first time'.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to incorporate the suggested clarifications and analyses.","responses":[{"response":"The referee correctly identifies that the abstract (and, per the comment, the manuscript) omits these implementation details. We will revise the abstract to include a brief description of confidence extraction (sequence-level average log-probability) and the CDG voting formula. We will also add statistical significance tests with multiple-comparison corrections to the experimental results.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the central empirical claim (positive vs. declining confidence trajectories) and the performance gains of CDG voting cannot be evaluated because the manuscript provides no description of how token- or sequence-level confidence is extracted, no exact voting formula, and no statistical tests or controls for multiple comparisons across models and benchmarks."},{"response":"This is a fair criticism; the current manuscript does not contain calibration analysis, length/format ablations, or explicit discussion of LLM calibration literature. We will add a dedicated subsection with these elements in the revision.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the load-bearing assumption that reported confidence scores supply a reliable monotonic signal of trajectory correctness is not supported by any calibration analysis, ablation on superficial factors (length, formatting), or comparison against known poor calibration of LLMs on reasoning tasks."}],"tokens_in":1300,"tokens_out":322,"duration_ms":33575,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one or two things to know: the authors report that correct reasoning traces in LLMs show rising confidence over the course of generation while incorrect ones do not, and they turn this into a new voting scheme called Confidence Dynamic Gain that they say outperforms standard methods on math problems.\n\nWhat the paper does well is run the method on a decent range of models and benchmarks. They evaluate on DeepSeek-R1, gpt-oss, Gemma-3, and Qwen-QwQ using AIME24, AIME25, HMMT25, and BRUMO25. That covers recent contest problems and multiple architectures, which is better than single-model tests. Mentioning code release is also useful for reproducibility. The fact that they claim theoretical insights suggests they tried to go beyond pure empirics.\n\nThe soft spots are around the core assumption. The pattern and the voting rule both depend on the model's confidence scores serving as a reliable signal of whether the current trajectory will end up correct. LLMs are typically poorly calibrated, especially on hard reasoning, so confidence can reflect things like how likely certain tokens are or how long the answer is rather than actual correctness. The abstract does not explain the precise extraction method or any diagnostics they ran to rule out those confounds. If the full paper has only basic comparisons without those checks, the gains could be less robust than presented.\n\nThis is aimed at people doing inference optimization for LLMs on reasoning tasks. Someone already using repeated sampling for answer selection would see direct applicability in the CDG rule if it works.\n\nThe work shows honest engagement with the problem of answer selection and presents falsifiable claims, so it deserves peer review. Referees can focus on the details of the confidence calculation and whether the theoretical section actually supports the empirical pattern.","headline":"The paper reports a rising-confidence pattern for correct LLM reasoning traces and proposes CDG voting on top of it, but the reliability of the confidence signal needs checking.","tokens_in":2312,"tokens_out":439,"would_cite":false,"duration_ms":38666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"LLM reasoning traces that reach correct answers tend to gain confidence over steps while wrong traces lose it.","keywords":["confidence dynamics","LLM reasoning","inference optimization","answer selection","chain-of-thought","model uncertainty","voting methods","mathematical benchmarks"],"falsifier":"A large set of examples in which many correct final answers come from trajectories whose confidence decreases or many incorrect answers come from trajectories whose confidence increases would falsify the reported pattern.","tokens_in":2582,"feed_emoji":"📈","tokens_out":623,"duration_ms":32504,"temperature":0.7,"pith_summary":"The paper shows that correct reasoning paths in large language models usually increase in reported confidence as they unfold, whereas incorrect paths show flat or dropping confidence. The authors turn this pattern into a voting rule called CDG that weighs answers according to the direction of their confidence change rather than final score alone. Experiments on math benchmarks with several model families report higher accuracy than standard repeated-sampling baselines. A reader would care because the method extracts a usable signal from uncertainty that already exists inside the model, without extra training or new data.","feed_headline":"Correct LLM paths gain confidence while wrong ones lose it","feed_subtitle":"A voting rule that tracks rising or falling confidence along reasoning traces improves answer selection on math benchmarks.","key_machinery":"Confidence Dynamic Gain (CDG) voting, which scores each sampled reasoning trace by the net change in model confidence across its steps and uses that score for final answer selection.","core_discovery":"Correct answer traces tend to exhibit confidence improvement over time (positive confidence gain), while incorrect traces show attenuated or declining confidence as reasoning proceeds. The authors introduce Confidence Dynamic Gain (CDG) based voting that incorporates the full trajectory of confidence evolution along the reasoning chain and report substantial gains over baselines on AIME24/25, HMMT25, and BRUMO25 across four model families, supported by theoretical analysis of the observed pattern.","pith_inferences":["If the pattern generalizes, prompts could be designed to make confidence trajectories even steeper for easier filtering.","The same dynamic might appear in non-mathematical domains such as code or scientific reasoning and could be tested directly.","CDG could be combined with existing techniques like majority voting to produce hybrid selection rules."],"forward_implications":["CDG voting improves answer selection accuracy when multiple reasoning traces are sampled at inference time.","The same confidence-trajectory signal works across different open-source model families on mathematical reasoning tasks.","Using the direction of confidence change supplies a stronger discriminative signal than endpoint confidence alone.","The method requires no model retraining and can be added to existing sampling pipelines."],"fun_headline_variants":["LLM confidence climbs on correct reasoning paths","Declining confidence marks incorrect LLM traces","Confidence dynamics distinguish right and wrong answers","Positive gain signals correct LLM reasoning traces","Right traces build confidence while wrong ones fade"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The model's reported confidence scores give a reliable, roughly monotonic signal of whether the current reasoning path is heading toward a correct answer.","fun_headline_variants_meta":{"raw":{"variants":["LLM confidence climbs on correct reasoning paths","Declining confidence marks incorrect LLM traces","Confidence dynamics distinguish right and wrong answers","Positive gain signals correct LLM reasoning traces","Right traces build confidence while wrong ones fade"]},"model":"grok-4.3","cost_usd":0.005455,"raw_usage":{"total_tokens":2614,"prompt_tokens":649,"num_sources_used":0,"completion_tokens":53,"cost_in_usd_ticks":54549500,"prompt_tokens_details":{"text_tokens":649,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1912,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":649,"tokens_out":53,"duration_ms":22748,"temperature":1.0,"reasoning_tokens":1912,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T11:09:28.509239+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A large set of examples in which many correct final answers come from trajectories whose confidence decreases or many incorrect answers come from trajectories whose confidence increases would falsify the reported pattern.","supporting_citations":[],"review_version":1}