{"id":"1c953721-43ff-41c5-b9fc-c16712bb764f","arxiv_id":"2607.08665","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An online resample-or-reroute policy that allocates each unit of a per-query budget by estimated marginal correctness per unit cost attains a better cost–quality Pareto front than single-route, cascade, and best-of-K baselines on four multi-draw LLM pools.","lead":"This paper treats resampling one LLM and switching to another as competing ways to spend a fixed per-query budget, and proposes a simple online policy that picks the next action by estimated correctness per unit cost. For multi-model serving systems, that framing can cut cost on easy queries and raise accuracy when models specialize.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The imperfect-verifier ablation still feeds ground-truth labels into allocation, so residual RoR gains at low q may not transfer to a fully noisy verifier.","rationale":"The reader correctly isolated the single softest load-bearing assumption: the ablation keeps perfect labels inside the allocation loop while only degrading final selection. That gap directly undercuts the transfer of the strongest empirical claim (verifier-gated Pareto gains, especially the +24.8-point GPQA margin). The paper already supplies the right negative evidence elsewhere (agreement collapses on GPQA; gains shrink with q), so the concern is not fatal; it simply means the residual low-q advantage is measured under an optimistic guidance signal. Keeping the verdict CONDITIONAL is therefore appropriate; the concrete noisy-w_im replay would either close the gap or confirm that the residual lead is an artifact. No stronger internal inconsistency appears: the problem formulation, the recoverability grounding, the multi-benchmark tensors, and the real-price / agreement robustness checks are all coherent and useful. The concern is precisely the one the reader flagged, so agreement is full.","tokens_in":15611,"tokens_out":643,"duration_ms":5847,"concrete_test":"Re-run the full q-sweep of Figs. 2–3 and the matched-budget numbers of Table V, but replace every w_im increment with a noisy observation: with probability q accept the true label, with probability 1-q flip it (or sample from a calibrated false-positive/false-negative rate). If the GPQA residual of RoR over budget-aware best-of-K at q=0.6 shrinks below ~3 points or disappears, the verifier-gated Pareto claim does not transfer to a fully noisy verifier.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that RoR attains a favorable cost–quality Pareto front, with largest gains on heterogeneous GPQA (0.892 vs 0.644 best-of-K) that are verifier-gated, rests on the imperfect-verifier ablation of Section VI–VII. That ablation degrades only final selection: “the per-draw guidance signal (the verified counts w_im in the belief update) still uses the recorded correctness labels, so the ablation isolates the effect of a degraded final-selection signal while the allocation signal is held fixed.” Consequently ˆp_im (Eq. 2) and the marginal-gain rule (Eq. 3 / Algorithm 1) continue to receive perfect w_im even at q=0.6. A real imperfect verifier would corrupt both selection and the online belief that decides whether to resample or reroute. The residual GPQA lead at q=0.6 (0.538 vs 0.444) and the claim that “gains are verifier-gated” are therefore measured under an optimistic allocation oracle that a deployed system does not have. The agreement-verifier replay (Table VIII) is only a partial check: it is label-free but still uses a deterministic consensus rule rather than a noisy continuous score that would also scramble the belief update.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This paper formulates budget-aware test-time model selection for LLMs: under a fixed per-query cost budget and an imperfect verifier, allocate each unit of budget between resampling a committed model and rerouting to an alternative so as to maximize expected correctness. It proposes an online resample-or-reroute (RoR) policy that greedily spends the next unit of budget on the action with highest estimated marginal correctness per unit cost (with a UCB variant and an oracle-allocation ceiling), and grounds the preference for resampling vs. rerouting in the recoverability asymmetry of a companion analysis. On newly regenerated multi-draw correctness tensors (k=30, seed-aligned) from an eleven-model open-weight pool over GSM8K, MATH-500, GPQA-Diamond, and HumanEval+, replay experiments report that RoR attains a favorable cost–quality Pareto front relative to single-route, one-commit router, budget-aware best-of-K, cascade, and random-allocation baselines, with the largest gains on heterogeneous GPQA (0.892 vs 0.644 best-of-K at matched mid budget). An ablation claims the gains are verifier-gated; robustness checks include a provider price vector, an agreement-based verifier, and a partial-test code verifier.","tokens_in":15947,"tokens_out":1564,"duration_ms":18912,"significance":"The framing of resampling and rerouting as competing uses of one explicit per-query budget under an imperfect verifier is a clean and practically relevant contribution that prior routing, cascade, and best-of-N lines treat separately. The empirical protocol is stronger than average for this area: seed-aligned multi-draw tensors, 50/50 train/test priors, 20 draw orderings, budget sweeps, sensitivity on the prior pseudo-count and train fraction, real-price replay, and two deployable verifiers (agreement; partial code tests). The CPU-only replay from released tensors is a genuine reproducibility strength. If the Pareto and verifier-gated claims hold under a fully noisy guidance signal, the work would give a deployable rule for when to spend budget on more samples of the same model versus switching models, with clear regime dependence (saturated vs. heterogeneous pools).","major_comments":[{"comment":"Section VI (Verifier paragraph) and the q-ablation in Section VII / Figs. 2–3: the parametric imperfect-verifier study degrades only final selection, while “the per-draw guidance signal (the verified counts w_im in the belief update) still uses the recorded correctness labels.” Consequently Eq. (2) and the marginal-gain rule (Eq. 3 / Algorithm 1) continue to receive perfect w_im even at q=0.6. A real imperfect verifier would corrupt both selection and the online belief that decides resample vs. reroute. The residual GPQA lead at q=0.6 (0.538 vs 0.444 best-of-K) and the central claim that “gains are verifier-gated” are therefore measured under an optimistic allocation oracle. Either re-run the ablation with noisy w_im (or a noisy continuous score feeding the belief), or substantially narrow the claim and treat Tables VIII–IX as the primary imperfect-verifier evidence.","section":null},{"comment":"Section IV.B and Algorithm 1: the marginal-gain estimator is fully specified only under early stopping, where \\Delta corr(a) reduces to ˆp_im and the rule becomes arg max ˆp_im/c_m. For the imperfect-verifier regime (q<1), early stopping is disabled and final selection is a soft mixture of “any correct” and “fraction correct.” It is not clear how \\Delta corr is estimated in that case, nor whether the same greedy ranking remains well-defined when the verifier score is continuous or noisy. Please state the general estimator used in the q<1 sweeps and confirm that the reported fronts use that estimator rather than the early-stopping reduction.","section":null},{"comment":"Section VI (Baselines) and Table V: the one-commit router’s cost is “proxied by one draw of the most accurate model” because the tensors store only the routed outcome. That makes the router appear as expensive as single-routing the best model (cost 32.0 on three of four benchmarks) and weakens the comparison to learned routers, which are often cheaper on easy queries. Either regenerate routed model identities so cost can be charged correctly, or present the router only as an accuracy reference at its true (unknown) cost and avoid matched-cost claims against it.","section":null}],"minor_comments":[{"comment":"Table I is useful but the “imperfect verifier” column for cascades/routers is a bit absolute; many cascades already use confidence gates. A short footnote clarifying the distinction (parametric imperfect verifier + joint resample/reroute budget) would help.","section":null},{"comment":"Figure 1 caption and Section VII: the low-budget GPQA exception (single-commit beats RoR below cost ~15) is important; consider marking that region on the plot or calling it out in the abstract’s “favorable Pareto” language so readers do not over-read dominance at all budgets.","section":null},{"comment":"Section IV.A: the claim that the policy is “insensitive to s over an order of magnitude” is only partly supported by Table VI (s=0.5 and s=2 match; s=8 costs 11.2 points on GPQA). Soften the wording to “stable for moderate s; large s over-anchors on heterogeneous pools.”","section":null},{"comment":"Notation: p_im is introduced as an unknown success probability, but the replay uses finite without-replacement draws from k=30. A sentence on finite-population bias (or that k=30 is treated as the generative process) would avoid a small mismatch between formulation and evaluation.","section":null},{"comment":"Related work: the reject-option and cascade citations are appropriate; a brief pointer to recent process-reward / verifier literature for LLMs would better situate the “plug in a concrete verifier” discussion in Section VIII.","section":null},{"comment":"Typos / polish: “test-timeresampling” and “budget-aware test-time model selection” spacing in the abstract; “Onsaturated” / “Onintermediate” / “Onhard” word-joining in Section VII; “arXiv:2607.08665v2” header is fine for preprint but should be cleaned for journal production.","section":null}],"recommendation":"major_revision","confidential_remarks":"The imperfect-verifier ablation flaw is real and load-bearing for the abstract’s “verifier-gated” claim, but the perfect-verifier Pareto results, real-price replay, and two deployable verifiers (agreement; partial code tests) are solid. I would not reject: a major revision that either re-runs allocation under noisy guidance or repositions the parametric q study as a selection-only stress test, with Tables VIII–IX as the primary imperfect-verifier evidence, would make this a strong methods paper. Dependence on the companion arXiv:2607.03436 is disclosed and appropriate; the present manuscript stands as an empirical/systems follow-up rather than a pure theory paper."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful thing here is operational, not theoretical: it treats one more sample of the committed model and the first sample of a new model as competing spends of a single per-query budget under an imperfect verifier, and gives a simple online rule (posterior mean of success probability over cost, with early stop). That framing is cleaner than the usual cascade-or-router split, and the companion recoverability asymmetry is used as grounding rather than as a substitute for evidence.\n\nWhat it does well is the replay. Seed-aligned 30-draw tensors over eleven open-weight models, four benchmarks of different difficulty, 50/50 priors, twenty draw orderings, budget sweeps, sensitivity on the prior strength, a real-price reweight, agreement verification, and a partial-test code verifier. On GPQA the mid-budget gap versus budget-aware best-of-K is large (0.892 vs 0.644); on saturated GSM8K the win is mostly cost. The paper is honest that gains shrink as the verifier degrades and that agreement fails on multiple-choice. The greedy rule is cheap and the UCB variant is correctly reported as no better. Citations cover routers, cascades, best-of-N, and reject-option work without padding.\n\nThe soft spot that matters is the one the stress note flags. In the parametric-q ablation, only final selection is degraded; the belief update still consumes ground-truth w_im. So residual advantage at q=0.6 is measured under an optimistic allocation signal a real noisy verifier would not supply. The agreement and base-test tables are better checks but do not fully close that gap. Minor issues: router cost is proxied by the best model, costs start as parameter counts (they do re-run under OpenRouter prices), and everything is offline replay. None of those sink the central claim for the tested pools.\n\nThis is for people building multi-model gateways or test-time compute allocators. It deserves a serious referee. I would engage with it, cite the formulation and the GPQA/verifier results, and ask for a fully noisy guidance ablation before treating the low-q numbers as deployment-ready.","headline":"Solid systems paper that unifies resampling and rerouting under one per-query budget; the Pareto evidence is real, but the imperfect-verifier ablation still feeds true labels into allocation.","tokens_in":16550,"tokens_out":534,"would_cite":true,"duration_ms":5802,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Under a fixed per-query budget and imperfect verifier, an online policy that spends each unit on either resampling the committed model or rerouting to another attains a better cost–quality tradeoff than single-commit routers, cascades, or p","keywords":["model routing","test-time compute","budget-aware inference","best-of-K sampling","LLM cascades","resample-or-reroute","cost-quality Pareto","imperfect verifier"],"falsifier":"Replay the same multi-draw tensors with a fully noisy guidance signal (so the belief counts themselves are corrupted by the same imperfect verifier used for final selection) and check whether RoR still dominates budget-aware best-of-K and cascade on the heterogeneous GPQA-style pool at matched mid budget.","tokens_in":16477,"feed_emoji":"⚖️","tokens_out":1025,"duration_ms":8276,"temperature":0.7,"pith_summary":"Serving a query with large language models is no longer just which model to call; it is how to spend a limited inference budget. Prior routing work commits once per query, while best-of-K work resamples one fixed model; neither treats those two moves as competing uses of the same per-query dollars. This paper formalizes budget-aware test-time model selection: given a cost limit and an imperfect verifier, decide at each step whether to draw again from a model already tried or to open a new model, so that expected correctness is maximized. The proposed resample-or-reroute policy scores every affordable action by estimated marginal correctness per unit cost and greedily takes the best. Replay on multi-draw correctness tensors from an eleven-model open-weight pool across four benchmarks shows the policy lies on a favorable cost–quality front relative to single-route, one-commit router, budget-aware best-of-K, cascade, and random baselines, with the largest lift on the most heterogeneous set. The advantage shrinks as verifier quality falls, and holds under a realistic price vector and certain deployable verifiers.","feed_headline":"Resample or reroute: one budget, better LLM answers","feed_subtitle":"An online policy spends each dollar on the next sample or a new model and beats pure routers and best-of-K","key_machinery":"Resample-or-reroute (RoR): a greedy (or UCB) online allocation that maintains a posterior success probability for each model on the current query and at every step chooses the affordable action—one more sample of a used model or the first sample of a new model—with the highest estimated increase in final correctness per unit cost, optionally early-stopping when the verifier accepts.","core_discovery":"When resampling a committed model and rerouting to an alternative compete for one per-query cost budget under an imperfect verifier, an online policy that always spends the next unit of budget on the action with highest estimated marginal correctness per unit cost attains a favorable cost–quality Pareto front versus single-route, one-commit-router, budget-aware best-of-K, cascade, and random-allocation baselines on the tested eleven-model pools, with largest gains on the most heterogeneous benchmark and with those gains gated by verifier quality.","pith_inferences":["Any serving stack that already has a process-reward model or execution harness can plug that signal into RoR without changing the allocation rule, turning an offline recovery guarantee into a live policy.","The same marginal-gain framing could be applied inside a single model’s token budget (continue generating vs. restart or switch decoding strategy), not only across models.","If future model catalogs keep growing more specialized rather than uniformly better, the value of the reroute branch will rise relative to pure resampling."],"forward_implications":["On heterogeneous model pools with a trustworthy verifier, systems should treat resampling and rerouting as interchangeable budget spends rather than separate mechanisms.","When the answer space is open-ended or a reliable partial checker exists (e.g., unit tests for code), test-time recovery of specialist advantage is available without a perfect oracle.","On near-saturated or homogeneous pools the simpler budget-aware best-of-K already captures most of the gain, so full RoR is unnecessary.","Agreement-based self-consistency is a usable free verifier for free-form answers but fails on multiple-choice tasks where wrong answers collide easily."],"fun_headline_variants":["Resample or reroute: spend each budget unit on highest marginal gain","Budget-aware RoR beats routers and best-of-K under imperfect verifiers","One cost budget: online policy chooses sample vs model switch for correctness","Resample-or-reroute policy lifts Pareto front on heterogeneous LLM pools","Verifier-gated gains: next budget unit goes to best marginal correctness"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The policy needs a usable verifier whose signal both guides which draw to take next and selects the final answer; without that, the advantage over simply resampling one strong model shrinks or disappears.","fun_headline_variants_meta":{"raw":{"variants":["Resample or reroute: spend each budget unit on highest marginal gain","Budget-aware RoR beats routers and best-of-K under imperfect verifiers","One cost budget: online policy chooses sample vs model switch for correctness","Resample-or-reroute policy lifts Pareto front on heterogeneous LLM pools","Verifier-gated gains: next budget unit goes to best marginal correctness"]},"model":"grok-4.5","effort":"low","cost_usd":0.004922,"raw_usage":{"total_tokens":1477,"prompt_tokens":889,"num_sources_used":0,"completion_tokens":81,"cost_in_usd_ticks":49220000,"prompt_tokens_details":{"text_tokens":889,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":889,"tokens_out":81,"duration_ms":4422,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T06:31:53.538957+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Replay the same multi-draw tensors with a fully noisy guidance signal (so the belief counts themselves are corrupted by the same imperfect verifier used for final selection) and check whether RoR still dominates budget-aware best-of-K and cascade on the heterogeneous GPQA-style pool at matched mid budget.","supporting_citations":[],"review_version":2}