{"id":"26ebb720-1d4c-423f-91df-f03a789709a5","arxiv_id":"2608.13087","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"The standard in-sample metric for test-time allocation in neural combinatorial optimization manufactures 2 to 3 percent phantom gains, but a split-sample correction removes them and preserves real gains under distribution shift.","lead":"A neural optimization audit shows that the standard way of measuring test-time sampling gains, deciding and scoring on the same samples, invents 2 to 3 percent gains that disappear when scored out of sample. The same fix keeps an 11 to 12 percent gain under distribution shift, so the method can separate phantom from real allocation value.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Out-of-sample d_split may be too downward-biased to support the claim that in-distribution allocation gains are zero; the null floor calibrates d_in, not d_split.","rationale":"The paper is a careful, transparent audit with strong support for the central phenomenon: the in-sample estimator d_in is biased upward and the instance-wise null floor shows that the customary procedure manufactures 2%-level gains where no gain exists by construction. The split-sample correction is a genuine improvement. The reader's conditional verdict is appropriate. The weakest point I see is not the exchangeability of the stored arrays, which is handled by random ordering, but the evidentiary role assigned to d_split in the in-distribution claim. d_split is introduced as an out-of-sample estimate and is honestly described as downward-biased, but the size of that downward bias is never measured. For the shifted workload, a downward-biased estimator that still shows 11-12% is evidence that a large real effect exists. For the in-distribution workload, however, the same downward bias means that d_split covering zero cannot by itself establish that the true gain is zero; it only establishes that the gain is below d_split's detection threshold. The abstract's 'that does not exist' and the conclusion's 'data where the true gain is zero' therefore overstate the evidence, since the true gain on the real in-distribution workload is not known to be zero. A synthetic positive-control calibration would settle this: if d_split reliably recovers gains of 1-2%, the negative claim is supported; if not, the paper's own 'no detectable headroom' language is the correct claim. The confirmatory endpoint-selection issue is real for transparency but is not the most load-bearing concern, because the confirmatory data use fresh seeds and thus selection on independent exploratory data does not, by itself, inflate the conditional Type I error of the selected endpoint. The unquantified downward bias of the estimator that carries the negative conclusion is more central to the paper's strongest claim.","tokens_in":12761,"tokens_out":11761,"duration_ms":148459,"concrete_test":"Build a synthetic positive-control workload with known true allocation gains of 1%, 2%, and 3% by constructing instances with known per-instance hardness (for example, a mixture of two instance families with known f_i curves), storing K=800 (Axis B) and K=1000 (Axis A) cost arrays per instance, and running the exact Section II-E d_split pipeline: split arrays in half, estimate marginal gains with convex minorant regularization, allocate greedily on the first half, evaluate on the second half, with N=50 and S/N=100. Compute bias-corrected bootstrap CIs for d_split at each known true gain. If d_split's CI excludes zero at true gains of 2% or less, the concern is refuted; if it covers zero, the in-distribution result should be reported only as 'not detectable at d_split's resolution', not as a gain that does not exist.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's negative conclusion rests on d_split covering zero as evidence that the true in-distribution allocation gain is zero or negligible. But d_split is explicitly biased downward: the allocation is decided from only half the stored array (400 trajectories on Axis B, 500 on Axis A), and the paper never quantifies the magnitude of that downward bias. The instance-wise null floor calibrates d_in, the in-sample estimator, not d_split. Without a mapping from d_split to the true gain, a confidence interval covering zero is compatible both with a true gain of zero and with a true gain of 1-2% that d_split simply cannot resolve. The abstract and conclusion phrase this as a gain 'that does not exist' and 'data where the true gain is zero', which is stronger than the 'no detectable headroom' language used in Sections II-E and VII. This does not weaken the core in-sample selection-bias finding, since the null floor shows that the customary procedure manufactures gains under a true-zero null, but it does weaken the stronger inference that the 2.2-2.6% in-distribution gains are entirely phantom. The positive shifted-workload result is less threatened because if d_split is downward-biased, the 11.5% figure is a conservative estimate, but the same estimator's detection power still needs to be calibrated before the negative claim is stated as a zero effect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper measures whether non-uniform instance-wise allocation of a fixed test-time sample budget improves neural combinatorial optimization (NCO) solvers, and audits the measurement procedure itself. It defines an allocation gain d as the relative improvement of an oracle allocation over uniform allocation, evaluated through an in-sample estimator d_in (allocation decided and evaluated on the same stored samples) and an out-of-sample estimator d_split (allocation decided on one half of the stored array, evaluated on the other). On in-distribution TSP-100 workloads, d_in shows gains of 2.2-2.6% across POMO, AM, and SymNCO with confidence intervals excluding zero, while d_split is indistinguishable from zero. The paper constructs an instance-wise null where the true gain is zero by construction and shows that the in-sample procedure manufactures gains of the same order; the resulting noise floor is reported as roughly invariant to the number of stored samples per instance and to the number of instances. Under a distribution-shifted workload (50% clustered TSP-100), a pre-registered confirmatory experiment finds d_split gains of 11.5% for AM and 12.0% for SymNCO, with a POMO negative control showing -0.3%. An exploratory budget-charged probe policy retains 3.4-4.6% at the registered composition. The paper provides a correction procedure, a reporting checklist, and releases data, code, and the full pre-registration record.","tokens_in":12916,"tokens_out":3977,"duration_ms":42037,"significance":"The paper makes a useful and carefully controlled methodological contribution. The offline replay setup is a genuine strength: all policies are compared on identical stored samples, removing run-to-run noise, and the separation of decision and evaluation samples is enforced cleanly. The pre-registration, negative control, and full release of arrays, scripts, and amendment records are exemplary and enable independent verification. The in-sample selection-bias finding, calibrated against an instance-wise null, is a new and important caution for test-time compute allocation in NCO and parallels the optimizer's-curse literature. The distribution-shift experiment is well designed to show that the correction discriminates rather than merely deflates. The main weakness is that the negative in-distribution conclusion is stated more strongly than the d_split estimator's precision can support: the paper itself notes that d_split is downward-biased, and a confidence interval covering zero for a downward-biased estimator does not establish that the true gain is zero. This is a fixable overreach, either by calibrating detection power or by softening the language.","major_comments":[{"comment":"The statement that the 2.2-2.6% in-distribution gains are gains that 'do not exist' or that the data have 'true gain zero' is stronger than the d_split evidence can support. The paper explicitly says in §II-E that d_split is biased downward because the allocation is decided from only half the stored array, and Limitation 4 reiterates that the decision uses 500 samples versus an evaluation budget of 100. A confidence interval covering zero for a downward-biased estimator is compatible with a true gain of zero but also with a true gain of roughly 1-2% that d_split cannot resolve. The abstract and conclusion phrase this as an effect that does not exist, whereas Sections II-E and VII use the more defensible 'no detectable headroom' language. Please either calibrate the detection power of d_split (for example, by injecting known amounts of heterogeneity into synthetic instances and measuring the recovery rate of d_split) or revise the abstract and conclusion to state that the gain is not detectable at the precision of d_split. This is load-bearing for the negative in-distribution claim, which is a central result of the paper.","section":"§II-E, §VII (Limitation 4), §VIII"},{"comment":"The offline replay estimator assumes that the stored cost arrays are exchangeable under random permutation, so that prefix minima under a random ordering estimate the expected best-of-k. This is plausible for stochastic rollouts on Axis A, but less obvious for Axis B, where the 800 stored costs come from 100 deterministic start-node trajectories each multiplied by 8 dihedral augmentations. If, for example, the eight augmentation copies of the same tour are highly correlated, or if the start-node trajectories have order-dependent correlations, then the replayed prefix minima may not correspond to the expected best-of-k of the actual multi-start procedure. Since d_in, d_split, and the instance-wise null all operate on these replayed arrays, any bias in the replay step propagates to all three. Please provide an empirical validation, e.g., compare the replay-based best-of-k curve against fresh online rollout best-of-k on a small set of instances, or otherwise justify exchangeability from the construction.","section":"§II-B, §II-D"}],"minor_comments":[{"comment":"The claim that the noise floor is 'flat within Monte-Carlo error' is not accompanied by estimates of that error. Please report standard errors or confidence intervals for the floor values in Tables II and III, or state the number of Monte-Carlo repetitions used, so the scale-invariance claim can be checked.","section":"§III-D, Tables II-III"},{"comment":"The header 'OOD shared_split' appears to be missing a space; it should read 'OOD share'. Also, 'd_charged(probe m = 20 charged)' is redundant; 'd_charged(probe m = 20)' would be clearer.","section":"Table VII"},{"comment":"The appendix is labeled 'APPENDIXB PRE-REGISTRATIONTIMELINE' and then immediately repeated as 'PRE-REGISTRATIONTIMELINE'; please fix the duplicate heading.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually transparent and well executed; the major issue is a mismatch between the strength of the negative conclusion and the precision of the d_split estimator. If the authors add a detection-power calibration or soften the zero-gain language, the paper would be a strong methodological contribution. The pre-registration and data release set a good example for the field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nYou should know about this paper if you care about test-time compute or NCO evaluation. The headline result — that the standard practice of deciding and evaluating an allocation on the same stored samples manufactures 2.2–2.6% gains when the true gain is zero — is real, and the evidence for it is unusually solid. The instance-wise null is a clean construction, the offline replay forces all policies to compare on identical samples, and the negative control on POMO is a nice touch. The paper is also exemplary in its transparency: it reports both in-sample and out-of-sample estimates, releases all arrays and code, pre-registers the confirmatory experiment, and even retracts its own earlier finding on the noise floor when it turned out to be an artifact of the maximum definition.\n\nThe main soft spot is the language. The abstract and conclusion say the in-distribution gain 'does not exist' and that the data have 'true gain zero.' What the experiment actually shows is that the out-of-sample estimator d_split covers zero. But d_split is biased downward — the allocation is decided from half the data — and the paper never calibrates that bias. So the confidence interval covering zero is compatible with a true gain of a couple percent that d_split simply cannot resolve. The categorical 'does not exist' overreaches; 'no detectable headroom within our detection limit' is what the data support. This doesn't hurt the in-sample bias finding, which is calibrated against the null, but it should be fixed in revision.\n\nThe second soft spot is the selection of confirmatory arms. The exploratory pass produced 12 cells; the two that were then pre-registered came from that pass, and the 10 negative cells are not shown. The new seeds and declared endpoints help, but the 11.5% CI is conditional on having chosen that cell. Reporting the full exploratory table would make the selection transparent.\n\nOne concern from the stress-test note doesn't hold up: the offline replay for Axis B. Randomizing the order of a fixed pool of deterministic trajectories makes the prefix min an unbiased estimate of the best of a random subset, so the exchangeability premise is sound.\n\nMy recommendation: send it to peer review, not desk reject. The core result deserves a serious referee, and the paper is well within the bar for a methodological contribution. The revision should soften the zero-gain claim, show the exploratory table, and ideally run the null through the split estimator to calibrate its power. I would use the checklist when I next evaluate test-time compute results.","headline":"Careful audit with a strong core result; the 'zero gain' language is overstrong and the confirmatory arms were selected, but this deserves peer review.","tokens_in":13535,"tokens_out":4802,"would_cite":true,"duration_ms":48614,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that the usual in-sample way of measuring test-time allocation gains manufactures 2.2-2.6% phantom gains on data where the true gain is zero, and that the same correction that removes them preserves real 11-12% gains…","keywords":["neural combinatorial optimization","test-time compute allocation","selection bias","optimizer's curse","best-of-k sampling","pre-registration","TSP-100","out-of-sample evaluation"],"falsifier":"Re-run the in-distribution audit with the allocation decided on stored samples but evaluated on freshly drawn independent samples from the same decoding distribution, at large N and K; if the out-of-sample gain exceeds the calibrated floor and replicates across new seeds, the claim that in-distribution allocation gain is zero would fall. Alternatively, show the noise floor dropping below, say, 0.5% as K or N grows far beyond the tested ranges, which would falsify the paper's scale-invariance claim.","tokens_in":12440,"feed_emoji":"🎲","tokens_out":6386,"duration_ms":59456,"temperature":0.7,"pith_summary":"This paper asks whether giving some test-time sample budgets more samples than others buys anything for neural combinatorial optimization, and whether the usual way of measuring that is trustworthy. It argues that the standard in-sample measurement—deciding and evaluating the allocation on the same stored samples—manufactures gains of 2.2-2.6% on data where the true gain is zero, and that this phantom gain does not shrink with more samples or more instances over the tested ranges. It then shows that reporting the gain out of sample removes the phantom gains on in-distribution workloads while preserving a real 11-12% gain under distribution shift for two of three solvers. A pre-registered confirmatory experiment, a negative control, and a budget-charged exploratory policy support the discrimination. The takeaway for a practitioner is that uniform allocation is an adequate default in distribution, but out-of-distribution allocation is worth measuring.","feed_headline":"In-sample scoring fakes 2.2-2.6% allocation gains","feed_subtitle":"The same out-of-sample correction wipes phantom gains and preserves a real 11.5% gain under shift.","key_machinery":"The central object is offline replay of stored sample arrays: for every instance, all sampled solution costs are stored, and any allocation policy is evaluated by taking the minimum of the first $k_i$ entries of a randomized permutation of that instance's array. This forces all policies to be compared on literally the same samples and makes the audit affordable. The two estimators are $d_{\\mathrm{in}}$ (allocation decided and evaluated on the same array) and $d_{\\mathrm{split}}$ (allocation decided on one half of the array, evaluated on the other); $d_{\\mathrm{in}}$ is biased upward by selection, while $d_{\\mathrm{split}}$ is the out-of-sample estimate. The calibration instrument is the instance-wise null: resampling with replacement from a single instance's array creates exchangeable synthetic instances with zero true allocation gain, and the distribution of per-source 95th percentiles defines the noise floor. Convex regularization—the greatest convex minorant of each estimated curve—restores the non-increasing marginal gains that greedy allocation requires.","core_discovery":"The central discovery is that the conventional evaluation protocol for test-time budget allocation in neural combinatorial optimization is dominated by selection bias: the allocation is chosen by maximizing over noisy per-instance estimates of best-of-$k$ curves, and then scored on the same realized samples, so the reported gain is the optimizer's curse in action. On uniform TSP-100 with three pretrained solvers, this in-sample procedure reports gains of 2.2-2.6% with confidence intervals excluding zero, while the same gains measured out of sample are 0.457, 0.015, and -0.512 percent, all indistinguishable from zero. The paper constructs an instance-wise null in which the true gain is zero by construction, and calibrates the in-sample estimate against it; the floor is flat in the number of stored samples and the number of instances over the tested ranges. Under distribution shift, the same out-of-sample correction leaves an 11.5% gain for AM and 12.0% for SymNCO at equal evaluation budget with the signal cost not charged, while a negative control solver shows no gain. The paper's own headline number is always stated with that signal-cost qualification, and an exploratory probe policy that charges a 20-sample probe retains 3.4% and 4.6%.","pith_inferences":["The same split-sample audit would likely change reported gains in other best-of-$k$ test-time compute settings, such as repeated sampling from large language models, where allocation decisions and evaluations often share the same rollout data.","The flatness of the noise floor suggests the bias is driven by per-instance estimation noise rather than sample size, which would imply that bigger experiments cannot rescue in-sample selection.","Because the budget-charged probe retains most of its headroom at low out-of-distribution shares, an adaptive policy that spends little when instances look in-distribution could be a practical next step.","The residual gain beyond the frozen distribution-label baseline hints that per-instance statistics carry information not contained in the shift label; a learned predictor of marginal sample value is a natural extension."],"forward_implications":["In-sample allocation gains at the 2% level should not be trusted unless the allocation is decided and evaluated on disjoint samples.","The noise floor must be computed per solver and configuration; it varies by roughly a factor of two across the three solvers here, so borrowing another paper's floor is not valid.","On in-distribution homogeneous workloads, uniform allocation is an adequate default within the detection limit of this audit.","Under distribution shift, allocation guided by held-out sample statistics is worth measuring and works without a distribution label; a deployable probe policy retains 3-5% at a 50:50 composition.","Increasing sample count or instance count did not shrink the phantom gain over the tested ranges, so scale is not a substitute for out-of-sample correction."],"supporting_citations":[{"why":"Supplies the Attention Model checkpoint used as one of the three pretrained solvers and defines the stochastic sampling inference mode on Axis A.","marker":"[1]"},{"why":"Supplies POMO, whose n-start times eight-augmentation protocol defines the finite pool on Axis B.","marker":"[2]"},{"why":"Supplies the SymNCO checkpoint used as the third solver and replication arm.","marker":"[13]"},{"why":"The optimizer's curse is the conceptual mechanism the paper quantifies in the allocation setting.","marker":"[3]"},{"why":"Provides the post-selection inference view of why maximizing over noisy estimates biases reported values.","marker":"[4]"},{"why":"Reality-check calibration is the conceptual ancestor of the instance-wise null floor.","marker":"[5]"},{"why":"Superior predictive ability testing is the other calibration ancestor of the null construction.","marker":"[6]"},{"why":"Difficulty-aware test-time compute allocation in language models is the neighbouring result the paper's audit is positioned against.","marker":"[7]"}],"fun_headline_variants":["Allocation gains vanish out of sample: 2.2-2.6% to zero","Selection bias inflates NCO budget gains; real gain only under shift","Phantom allocation gains: in-sample vs out-of-sample","NCO budget gains: oracle bias, then a real 11.5% under shift","Auditing test-time budget: fake gains, real shift gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The audit assumes that randomized prefix minima of stored per-instance cost arrays are unbiased estimates of true expected best-of-$k$, so the stored arrays faithfully emulate online sampling and are exchangeable under random ordering.","fun_headline_variants_meta":{"raw":{"variants":["Allocation gains vanish out of sample: 2.2-2.6% to zero","Selection bias inflates NCO budget gains; real gain only under shift","Phantom allocation gains: in-sample vs out-of-sample","NCO budget gains: oracle bias, then a real 11.5% under shift","Auditing test-time budget: fake gains, real shift gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1540,"prompt_tokens":1228,"completion_tokens":312,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":844,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":844,"tokens_out":312,"duration_ms":3334,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:01:28.452545+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the in-distribution audit with the allocation decided on stored samples but evaluated on freshly drawn independent samples from the same decoding distribution, at large N and K; if the out-of-sample gain exceeds the calibrated floor and replicates across new seeds, the claim that in-distribution allocation gain is zero would fall. Alternatively, show the noise floor dropping below, say, 0.5% as K or N grows far beyond the tested ranges, which would falsify the paper's scale-invariance claim.","supporting_citations":[{"cited_title":"POMO: Policy Optimization with Multiple Optima for Reinforcement Learning","cited_arxiv_id":"2010.16011","evidence_quote":"Supplies POMO, whose n-start times eight-augmentation protocol defines the finite pool on Axis B."},{"cited_title":"Inference on winners,","cited_arxiv_id":null,"evidence_quote":"Provides the post-selection inference view of why maximizing over noisy estimates biases reported values."}],"review_version":1}