{"id":"2153ba60-9c5c-4ed7-815f-62e932018063","arxiv_id":"2607.05391","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.","lead":"This paper turns LLM judges into continuous verifiers by averaging scoring-token probabilities instead of taking a single discrete score, then scales that signal with finer score bins, repeats, and split criteria. The method improves best-of-N selection and dense rewards for coding, robotics, medical agents, and RL without training a new reward model.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The central claim rests on scoring-token logits being a calibrated proxy for trajectory correctness/progress; if they mainly track surface form or prompt style, ranking, VOC, and RL gains would not transfer as claimed.","rationale":"The reader correctly isolates the premise that enters at Eq. 3.1–3.2, §4.3 criteria design, and App. B.6: scoring-token mass as a calibrated quality/progress proxy. That premise is necessary for treating verification as a general-purpose scaling axis that transfers to ranking, VOC, and RL without training. Empirical ablations (granularity SNR, tie-rate reduction, multi-domain gains, high robotics VOC) give real support, so the paper is not internally inconsistent and does not warrant REJECT; the risk is medium and already reflected in CONDITIONAL. My concrete test sharpens the same concern with a controlled criteria ablation rather than inventing a separate flaw (e.g., leaderboard compute matching is a real but secondary comparison issue the paper partially discloses via Pass@1/oracle). No stronger load-bearing failure is evident from the manuscript alone; fuller public artifacts would still help, but that does not change the verdict.","tokens_in":28876,"tokens_out":727,"duration_ms":30088,"concrete_test":"On a fixed Terminal-Bench V2 pair set (same as Fig. 4 / SNR Table 1), recompute pairwise accuracy and SNR(G=20) under three prompt ablations: (i) domain criteria as published, (ii) a single generic “rate overall quality 1–20” rubric, (iii) deliberately mismatched criteria (e.g., medical safety language on coding trajectories). Hold G, K, and the candidate pool fixed. If accuracy/SNR under (ii)–(iii) stay within ~2 points of (i), the logits are largely criteria-independent surface cues and the calibration premise fails; a large drop supports the paper’s claim that criteria-driven verification is doing the work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 3.1 defines the continuous reward as the expectation of φ(v_g) under p_θ(v_g | x, c, τ) over hand-written criteria c, then uses those scores for pairwise ranking (Eq. 3.2 / PPT), Value-Order Correlation (§6), and dense RL rewards (Eq. 7.1–7.2). The load-bearing premise is that this probability mass is a domain-general, calibrated belief about correctness/progress rather than a reflection of trajectory surface features (length, log verbosity, tool-call polish), positional artifacts, or how confidently the model phrases a hedged judgment. The paper’s own query-optimize case (Table 2, App. B.4) shows the verifier correctly spotting a methodological failure yet expressing it in graded language (“slightly cleaner,” “marginally more direct”), so the continuous expectation can still rank well while encoding stylistic confidence. Criteria decomposition (§4.3) and the two-stage closed-model workaround (App. B.6) further entangle the signal with human rubric design and a second model’s logprobs. If the logits are not primarily correctness-calibrated, the claimed verification-scaling axis (G/K/C), SOTA trajectory selection, VOC progress monitoring, and RL sample-efficiency gains would be fragile outside the reported pools and prompts.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes LLM-as-a-Verifier, a training-free verification framework that scores agent trajectories by taking the expectation of a scalar map over the model’s scoring-token probability mass (Eq. 3.1), rather than collapsing to a discrete argmax score as in standard LM judges. It argues that this continuous formulation unlocks three complementary verification-scaling axes—score granularity G, repeated evaluation K, and criteria decomposition C—and introduces Probabilistic Pivot Tournament (PPT) to select among N candidates at O(Nk) pairwise cost. Empirically, the method reports improved pairwise verification accuracy under controlled G/K/C budgets (Fig. 4, Table 1, Fig. 7), competitive or state-of-the-art trajectory selection on Terminal-Bench V2 (86.5%), SWE-Bench Verified (78.2%), RoboRewardBench (87.4% preference accuracy), and MedAgentBench (73.3%), Value-Order Correlation as a progress proxy, and denser rewards that improve sample efficiency of DSRL-SAC on LIBERO and GRPO on MATH (Fig. 9).","tokens_in":29337,"tokens_out":991,"duration_ms":23934,"significance":"If the results hold under broader scrutiny, the paper makes a useful contribution to test-time scaling and agent evaluation: it reframes verification quality as something that can be improved by compute and prompt structure without reward-model training, and it demonstrates transfer across coding, robotics video, and medical agent settings with a single probabilistic recipe. Strengths include controlled scaling plots, an explicit SNR decomposition for granularity (Table 1), a concrete case study with tie-rate analysis (Table 2), a budget–accuracy characterization of PPT (Table 9), multi-benchmark selection results with Pass@1/oracle context (Table 3), and dual use of the same signal for progress monitoring and RL shaping. The practical artifacts (Claude Code/Codex-style extension, harness generalization in Appendix B.1, closed-model two-stage workaround in B.6) increase the work’s utility beyond a pure leaderboard claim.","major_comments":[{"comment":"§3.2 Eq. (3.1)–(3.2) and §4.1: The central interpretation—that continuous scoring-token expectations yield more calibrated correctness comparisons—is supported by SNR growth, lower tie rates, and accuracy gains, but the manuscript does not adequately rule out surface-form confounds (trajectory length, log verbosity, tool-call polish, formatting). The query-optimize case (Table 2, App. B.4) shows the model can identify a real methodological failure while still expressing it in hedged language, which is consistent with either calibrated belief or stylistic confidence. A load-bearing addition would be matched-surface ablations or partial correlations of R(x,τ) with length/verbosity after conditioning on success/failure; without this, the claim that verification is a distinct scaling axis (vs. a better readout of existing judge biases) remains incompletely stress-tested.","section":null},{"comment":"§4.3 and the multi-domain experiments in §5: Criteria decomposition is presented as a general scaling axis, yet the concrete factors (Specification/Output/Errors for code; domain-specific criteria elsewhere) are hand-designed. Fig. 4 (right) shows ensemble gains, but there is no sensitivity analysis to criterion wording, number, or quality, nor a clear protocol for constructing criteria in a new domain. Because SOTA numbers use G=20, K=8, and the three-criterion setup, the paper should quantify how much of the reported selection accuracy is attributable to rubric engineering versus the continuous formulation and PPT; otherwise the “general-purpose / no additional training” claim overstates plug-and-play generality.","section":null},{"comment":"§5 and Table 3: Absolute “state-of-the-art” framing should be tightened. The fairest evidence is improvement over Pass@1 on the same candidate pools (e.g., Terminal-Bench 83.1%→86.5%, SWE-Bench 76.1%→78.2%, MedAgent 70.2%→73.3%), which is real but modest relative to oracle headroom. Leaderboard comparisons mix harnesses, proposal models, and N; SWE-Bench further uses a heterogeneous three-model pool, so selection partly includes cross-model routing. Please report selection accuracy with uncertainty (bootstrap/task-level variance), fix N and harness when comparing to named baselines, and separate “best-of-N with our verifier” from “beats published single-trajectory leaderboard entries.”","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a careful empirical packaging of probability-based judging for long-horizon agent trajectories. The continuous expectation over scoring-token logits (Eq. 3.1), the three scaling knobs (granularity, repeats, criteria), and the Probabilistic Pivot Tournament are the usable pieces. They get real gains on Terminal-Bench, SWE-Bench Verified, RoboRewardBench, and MedAgentBench, plus VOC progress signals and denser SAC/GRPO rewards—without training a verifier.\n\nWhat is actually new is less the logit expectation itself (GPTScore, G-Eval, generative verifiers, TOPReward already live nearby) and more the coherent treatment as a verification-scaling axis for full agent trajectories across code, robots, and medical tools, plus a budgeted ranking algorithm that cuts O(N²) comparisons. The paper does this well: Fig. 4 and the SNR/tie-rate analyses are clean; the query-optimize case shows why continuous scores break the 88% tie rate of discrete 1–5 judges; Table 9 shows PPT tracking full round-robin with fewer pairs; robotics preference/MAE/VOC numbers beat trained RMs zero-shot. The RL plots are modest but directionally right.\n\nSoft spots, in proportion: novelty is incremental, not a new theory of verification. SOTA numbers are pool- and harness-dependent (Pass@1 vs oracle headroom is honest, but leaderboard comparisons still need careful matching). Criteria are hand-written; logprob access is a real deployment constraint (the two-stage GPT→Gemini workaround helps but is a patch). The stress-test worry—that logits track surface polish more than correctness—is fair to keep in mind, but the paper’s own evidence (SNR rising with G, zero ties, cross-domain transfer, failure-mode detection in the case study) makes it less load-bearing than a pure surface-form story would imply. Free parameters (G, K, C, k, λ, β) are ablated rather than hidden.\n\nWho this is for: people building best-of-N agent systems, robotics reward proxies, or dense RL feedback who want a training-free baseline with clear knobs. Math and citations look fine; related work is not airbrushed. I would send it to peer review. Worth engaging if you work on agent verification or test-time selection; not required reading if you only care about fundamental scaling laws.","headline":"Solid multi-domain systems paper: continuous score-token expectation plus G/K/C scaling and PPT ranking beat discrete judges and several trained robotics RMs, with real ablations—but novelty is packaging more than a new principle.","tokens_in":29879,"tokens_out":613,"would_cite":true,"duration_ms":10500,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Verification is a new scaling axis for language models: continuous scores from scoring-token logits, scaled on three knobs, select better agent trajectories without training a judge.","keywords":["LLM-as-a-Verifier","verification scaling","scoring-token logits","trajectory reward model","test-time scaling","dense rewards","agentic tasks","Bradley-Terry ranking"],"falsifier":"On a held-out set of trajectory pairs whose ground-truth correctness is known, replace continuous logit expectations with ordinary discrete argmax scores (or with random continuous scores of matched variance) while keeping the same ranking procedure; if the accuracy and progress-correlation gains disappear, the central claim fails.","tokens_in":29816,"feed_emoji":"✅","tokens_out":923,"duration_ms":7952,"temperature":0.7,"pith_summary":"The paper argues that deciding whether a solution is correct has been left behind while generation has scaled, and that verification itself can be scaled. Standard language-model judges collapse their scoring distribution into a single discrete token, which creates ties and weak discrimination among long agent trajectories. LLM-as-a-Verifier instead takes the expectation over the full distribution of scoring-token logits, producing continuous scores. That probabilistic formulation opens three practical knobs: finer score granularity, repeated evaluations, and decomposition into simpler criteria. With a budget-aware ranking procedure that concentrates comparisons on the most promising candidates, the same training-free verifier reaches new highs on coding, robotics, and medical agent benchmarks and supplies dense progress signals that speed up both off-policy and on-policy reinforcement learning.","feed_headline":"Verification scales: continuous scores pick better agent paths","feed_subtitle":"No new training needed: logit expectations plus three knobs beat discrete judges on coding, robots, and medical tasks","key_machinery":"The continuous reward of Equation 3.1: the average, over criteria and repeated evaluations, of the expected scalar value of the scoring-token distribution. Pairwise preferences follow from a Bradley–Terry model on those rewards; a Probabilistic Pivot Tournament then ranks N candidates in O(Nk) comparisons by first breaking positional bias with a random ring pass and then concentrating remaining budget on a small set of pivots.","core_discovery":"Verification quality improves systematically when continuous rewards are formed as the expectation over scoring-token logits and then scaled along score granularity, number of repeated evaluations, and criteria decomposition. Combined with a cost-efficient pivot tournament that converts those continuous scores into preference probabilities, the framework selects better trajectories at test time and supplies denser rewards for reinforcement learning, all without training a specialized reward model.","pith_inferences":["If scoring-token mass is truly calibrated, criteria decomposition may eventually be generated automatically per domain rather than hand-designed, turning the third knob into a learned object.","The same continuous signal could serve as a runtime monitor that freezes or rolls back an agent when the progress curve plateaus or declines, independent of final-outcome ranking.","Closed models that withhold logits may still participate via the two-stage handoff, which suggests a practical division of labor: frontier reasoning + open calibrated scoring.","Because the framework is training-free, gains should appear most clearly on domains where collecting preference data for a learned reward model is expensive or safety-critical."],"forward_implications":["Best-of-N selection for long-horizon coding, robotics, and medical agents can improve without any new reward-model training.","Live progress scores become available for monitoring and early abort of agentic systems.","Dense verifier rewards raise sample efficiency of both SAC-style and GRPO-style reinforcement learning.","Verification compute can be dialed independently of generation compute via the three scaling knobs and the pivot budget.","The same recipe transfers across text and multi-frame video once log-probabilities (or a two-stage open-model handoff) are available."],"fun_headline_variants":["Continuous logit expectations turn verification into a scaling axis","Score granularity and repeats lift agent solution ranking accuracy","No-train LLM verifier ranks better trajectories via continuous scores","Criteria decomposition plus logit scores densifies agent feedback","Verifier continuous scores improve test-time picks and RL efficiency"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a language model’s probability mass over scoring tokens, when prompted with hand-written criteria, is a calibrated, domain-general measure of trajectory quality rather than an artifact of prompt style or superficial cues.","fun_headline_variants_meta":{"raw":{"variants":["Continuous logit expectations turn verification into a scaling axis","Score granularity and repeats lift agent solution ranking accuracy","No-train LLM verifier ranks better trajectories via continuous scores","Criteria decomposition plus logit scores densifies agent feedback","Verifier continuous scores improve test-time picks and RL efficiency"]},"model":"grok-4.5","effort":"low","cost_usd":0.004946,"raw_usage":{"total_tokens":1449,"prompt_tokens":889,"num_sources_used":0,"completion_tokens":58,"cost_in_usd_ticks":49460000,"prompt_tokens_details":{"text_tokens":889,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":502,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":889,"tokens_out":58,"duration_ms":4434,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T07:01:25.730013+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out set of trajectory pairs whose ground-truth correctness is known, replace continuous logit expectations with ordinary discrete argmax scores (or with random continuous scores of matched variance) while keeping the same ranking procedure; if the accuracy and progress-correlation gains disappear, the central claim fails.","supporting_citations":[],"review_version":2}