{"id":"37e8b121-066e-4fa6-bb92-28af100ea59a","arxiv_id":"2608.00296","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A stabilized rank-gap variant of Leader Reward reduced Best-of-8 TSP-100 cost in all three paired seeds (7.7944 vs 7.8136), but the gain is decoder-specific and not statistically confirmatory.","lead":"This short experimental note tests a tweak to POMO training for the traveling-salesman problem: instead of emphasizing only the best trajectory, it ranks the top K sampled trajectories and trains on a stabilized version of that rank signal. In three paired training runs, the tweak lowered the realized best-of-8 cost on TSP-100 (7.7944 vs 7.8136), while remaining worse under the original decoding protocol.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central observation may be a selection artifact: K=8 and δ=0.01 are hand-set with no sensitivity analysis or preregistration, and 3/3 directional success has 1/8 probability under a null.","rationale":"The paper is transparent and self-aware, explicitly disclaiming unbiased estimation, universal superiority, and state-of-the-art claims. It provides the full per-seed data, the matched schedule, and an honest claim boundary. The reader's weakest assumption—that K=8 and δ=0.01 might be selection artifacts given the lack of preregistration and sensitivity analysis—is exactly the load-bearing concern. I agree with the conditional verdict: the observation is plausible but not confirmatory. The strongest technical alternative concern, that the stabilization's effectiveness could stem from the standardization rather than the rank-gap signal, is also acknowledged in the paper's claim boundary and does not change the empirical claim about the full recipe. The most direct threat to the central observation is the possibility that the two free parameters were tuned on the same test protocol, combined with the small seed count. A sensitivity sweep with additional seeds would settle this. I therefore recommend no change to the reader's CONDITIONAL verdict.","tokens_in":3950,"tokens_out":5475,"duration_ms":55515,"concrete_test":"Run a preregistered sensitivity study with the same training protocol and evaluation pipeline, varying K over {2,4,8,16} and δ over {0, 0.001, 0.01, 0.1} using the same three seeds (or ideally six), with the primary endpoint fixed as K=8, δ=0.01. If the sign and approximate magnitude of the advantage do not persist across neighboring settings and additional seeds, the original 3/3 result is likely an artifact of the chosen operating point.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the stabilized K=8 recipe lowers realized Best-of-8 cost in all three paired training seeds—depends on the assumption that K=8 and δ=0.01 were fixed a priori rather than selected after inspecting the same test protocol. The paper states this independence (Evaluation card; 'the recipe's success is not a selection artifact') but provides no external preregistration and no sensitivity analysis. With only three seeds, the probability of observing 3/3 positive differences under a true null of no effect is 1/8 = 0.125, which the authors themselves acknowledge via their 'six-seed testing floor.' Moreover, Table 2 shows the advantage is not unique to K=8: the gap is larger at K=2 and K=4, so the choice of K=8 is arbitrary relative to the effect. Without a sweep over K and δ, or a preregistered fixed setting, the observed 3/3 direction could be a cherry-picked operating point, and the paper provides no evidence to rule this out. This is not an internal mathematical inconsistency—the estimator identity is openly disclaimed—but a threat to the empirical validity of the headline observation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies a narrow modification of Leader Reward for POMO-based neural combinatorial optimization. Instead of giving extra weight only to the current leader trajectory, the proposed 'stabilized Best-of-K' recipe assigns weights based on rank-gaps among the top K trajectories, applies a floor and per-instance standardization, and uses the resulting weights in the policy-gradient loss. With the POMO architecture, the original 3,050-epoch schedule, and the TSP-100 test set held fixed, the authors report that the stabilized K=8 recipe yields a lower realized Best-of-8 cost than Leader Reward in all three paired training seeds (7.7944 versus 7.8136). They explicitly label the result as estimation-only and decoder-specific, acknowledge that Leader Reward is better at K=1 and under augmented-greedy decoding, and make no claim of unbiased estimation or state-of-the-art performance.","tokens_in":4194,"tokens_out":6814,"duration_ms":70440,"significance":"If the observed improvement is robust, the paper would provide a simple, budget-indexed alternative to Leader Reward that could improve deployment-time Best-of-K performance in NCO. The manuscript is unusually transparent: it reports all three seeds, all tested K rows, evaluation runtimes, an artifact manifest, and a clearly marked CLAIM BOUNDARY section. It also makes a falsifiable prediction — that the gain depends on the sampling budget — which is supported by the reported K-curve. The main limitations are the small seed count (three) and the lack of sensitivity analysis for the two free parameters (K and δ), which weaken the generality of the 'promising extension' conclusion.","major_comments":[{"comment":"The primary result is based on three training seeds, one half of the paper's own six-seed floor. The paper correctly notes that the minimum one-sided signed-rank p-value is 1/8 and that the BCa interval is fragile, but the abstract's verb 'lowers' and the Bottom Line's 'promising extension' go beyond a strictly descriptive reading. Please either add the additional three seeds required by the stated floor, or rephrase the central claim as 'in our three paired seeds the stabilized K=8 recipe produced lower realized Best-of-8 cost' and temper the conclusion accordingly.","section":"Evaluation card / Finding 2 (Table 1)"},{"comment":"The recipe fixes K=8 and δ=0.01. The paper asserts these were chosen independently of the evaluation, but provides no preregistration, protocol document, or sensitivity analysis to support that assertion. Table 2 shows that the advantage is actually larger at K=2 and K=4 than at K=8, which mitigates K-selection but does not eliminate it; no analogous check is provided for δ. Without a small sweep over δ (e.g., 0, 0.001, 0.01, 0.1), the reader cannot tell whether the observed improvement is due to the rank-gap signal or to the particular floor value. This is load-bearing for the conclusion that the recipe is 'promising' rather than an artifact of one operating point.","section":"Eq. (3)-(4) / 'The tested budget-indexed extension'"},{"comment":"The non-monotonic dependence of the advantage on K is not discussed. The gap is largest at K=2 and K=4, then shrinks monotonically through K=128. Since the paper posits a 'budget-indexed' training signal, it should explain why the effect peaks at small K and why K=8 is the emphasized operating point. Without this discussion, the selection of K=8 appears arbitrary, and the 'gain shrinking as K grows' narrative in the abstract is an incomplete description of the observed curve.","section":"Table 2 / Finding 3"}],"minor_comments":[{"comment":"The phrase 'lowers realized Best-of-8 cost' is stronger than the estimation-only boundary. Consider changing to 'was lower' or 'we observed lower' to align with the stated non-confirmatory status.","section":"Abstract / Bottom Line"},{"comment":"The indexing in the inner sum uses R_(m-1) for m from K to i; please clarify the relationship to the sorted rewards and why the cumulative sum starts at m=K rather than m=i+1. A short derivation or example would help.","section":"Eq. (3)"},{"comment":"The statement 'the remaining gap changes from 0.626% to 0.378%, a 39.7% reduction' is a nonstandard effect-size presentation because it expresses a percentage of the optimality gap rather than the relative difference between methods. The relative improvement is 0.247%; consider reporting this consistently.","section":"Finding 2"},{"comment":"The bands show 'one population standard deviation across seeds'. With only three seeds, a population standard deviation is not well-defined; please use the sample standard deviation or omit the band.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and appropriately cautious, and the authors have been unusually transparent about limitations and artifacts. The main concern is that the central 'promising extension' conclusion relies on an independence assumption that is not verifiable from the reported experiments, especially given only three seeds and no δ sensitivity. If the authors can add a δ sweep and either meet the six-seed floor or soften the conclusion to a purely descriptive observation, the paper would be acceptable. I do not see evidence of misconduct, but the statistical support is currently too thin for a confirmatory claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honest, well-scoped empirical note. The new thing is a stabilized rank-gap weight (Eq. 3–4) that extends Leader Reward, plus a clean controlled comparison across sampling budgets K and decoder protocols. The headline observation — 3/3 seeds improve Best-of-8 cost at K=8 (7.7944 vs 7.8136) — is plausible and the authors report all the numbers needed to check it, including the full K-curve and the augmented-greedy row where Leader Reward still wins.\n\nWhat it does well: the baseline reproduction is exact (7.7662 vs 7.766), the claim boundary is unusually explicit (no unbiased-estimator or SOTA claims; estimation-only; three seeds below their own six-seed floor), and the artifact package includes per-seed curves, runtimes, and checkpoints. That is more than most arXiv notes ship.\n\nSoft spots, in proportion: the central result rests on three paired seeds and on two hand-set hyperparameters (K=8, delta=0.01) with no sensitivity sweep or preregistration. The stress-test's selection-artifact concern is legitimate: choosing the headline K after seeing the same test protocol could inflate the apparent hit rate, and 3/3 under a null has probability 1/8. That said, the paper reports the entire K curve rather than cherry-picking a single row, and the advantage is present from K=2 to K=128, so the observation is not a lone point. Still, the delta floor is completely unexamined, and the absence of a vanilla-POMO or objective-faithful Max@K arm means we cannot tell which component (floor, standardization, rank gap) drives the gain. These gaps are disclosed, but they remain gaps.\n\nVerdict: this deserves a serious referee, not because the effect is large — it is a 0.25% cost reduction on one benchmark — but because the reproduction effort and transparency are exemplary and the K-dependence finding is useful for people building Best-of-K training variants. I would accept it for peer review with an expectation of revision: add more seeds (or clearly frame as preliminary), run a small sensitivity check on delta and K, and add one comparator arm. I would probably not cite it in my own work unless I were specifically working on POMO-style training extensions.","headline":"Transparent, narrow empirical note; plausible 3/3 gain at K=8 but under-powered and hand-set hyperparameters.","tokens_in":4711,"tokens_out":2462,"would_cite":false,"duration_ms":24619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stabilized K=8 rank-gap training signal lowers realized Best-of-8 TSP-100 cost in all three paired seeds, while leader-reward remains better at K=1 and under augmented-greedy decoding.","keywords":["neural combinatorial optimization","TSP-100","best-of-K training","leader-reward training","rank-gap credit assignment","policy gradient","sampling budget","multi-start neural architectures"],"falsifier":"Train the same two arms with six paired seeds under the same frozen protocol; if the stabilized Best-of-8 mean is not lower than leader-reward in at least five of six seeds, the reported direction fails to replicate. A grid over K near 8 and δ near 0.01 would also reveal whether the improvement is a narrow tuned peak or a plateau.","tokens_in":3779,"feed_emoji":"📉","tokens_out":7513,"duration_ms":67896,"temperature":0.7,"pith_summary":"The paper asks whether a training signal that rewards all trajectories that could become the best among K sampled candidates beats a binary leader-only signal when deployment draws exactly K independent trajectories. It changes only the trajectory-weight rule inside an existing multi-start neural combinatorial-optimization trainer, keeping architecture, schedule, and test set fixed. On TSP-100, the stabilized K=8 recipe lowers realized Best-of-8 cost in all three paired training seeds, by an average of 0.247%. The paper is explicit about the boundary: the gain contracts as K grows, reverses at K=1, and does not transfer to the original augmentation-based decoder, so this is an engineering observation about a deployment regime, not a claim of general superiority.","feed_headline":"Stabilized best-of-8 training trims route costs in all seeds","feed_subtitle":"On TSP-100, the gain is 0.247% on average and disappears for K=1 or augmented-greedy decoding.","key_machinery":"The central mechanism is the stabilized rank-gap weight: rewards from one instance's multiple starting points are sorted, trajectories below rank K receive zero weight, and trajectories at or above rank K receive credit proportional to a budget-weighted sum of gaps to earlier rewards, floored by a small fraction of the reward range and then standardized within the instance. This substitutes a graded, budget-indexed credit assignment for the sharp leader/non-leader split. The floor scale δ=0.01 and the chosen K=8 are the settings that carry the reported effect; the paper does not ablate them.","core_discovery":"The central discovery is that replacing the binary leader/non-leader trajectory weight with a rank-gap signal indexed by a sampling budget K produces a policy whose realized Best-of-8 cost is lower in every paired seed: 7.7944 versus 7.8136 for the leader-reward baseline, a 0.0193 average paired difference with a descriptive BCa interval of +0.0068 to +0.0257. The advantage appears for K between 2 and 128, is largest at small budgets, and disappears at K=1 and under augmented-greedy decoding. The authors emphasize that the recipe is not an unbiased gradient estimator and that the floor, standardization, and distinct-start training law prevent an exact Max@K interpretation.","pith_inferences":["A natural test the paper does not run is a sensitivity sweep over K and δ; if the improvement persists across neighboring budgets, the effect is structural, whereas if it peaks only at K=8, the 3/3 result could be tuning noise.","The per-instance standardization may itself be the active ingredient, since it changes the effective learning-rate scale; an ablation that removes only the floor, or only the standardization, would separate mechanism from recipe.","Editorially, the result is a useful reminder that training objective and deployment protocol do not need to match exactly: the rank-gap surrogate is trained with distinct starting nodes but evaluated on independent samples, and still transfers.","If a six-seed confirmatory study succeeds, the same recipe could be tested on harder routing problems (CVRP or FFSP) to see whether the small-budget regime benefit generalizes."],"forward_implications":["If the observation holds, deployment that samples a modest number of independent candidate trajectories can get a small but repeatable cost reduction from the same policy architecture by switching the training credit rule.","The advantage is specific to the sampled best-of-K regime: users should keep leader-reward training when they deploy one sample or an augmented-greedy decoder.","The reported gap to the optimal-cost reference shrinks from 0.626% to 0.378% at K=8, suggesting the rank-gap signal captures useful ordering information that leader-only credit discards.","The paper's own six-seed testing floor means the result should be treated as an estimate; a confirmatory run with six paired seeds is the direct next step.","Since the improvement contracts as K grows, the signal's value is tied to small-budget deployment, not to asymptotic best-of-K performance."],"fun_headline_variants":["Rank-based reward beats binary leader on TSP-100 Best-of-8","Stabilized Best-of-8 training trims costs in all three seeds","POMO tweak lowers TSP-100 costs, but only for sampled K>1","New training signal beats leader reward, but not at K=1"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that K=8 and δ=0.01 were fixed without tuning to the evaluation protocol and that three paired seeds adequately represent training randomness; if either assumption fails, the all-three-seeds improvement could be a selection artifact.","fun_headline_variants_meta":{"raw":{"variants":["Rank-based reward beats binary leader on TSP-100 Best-of-8","Stabilized Best-of-8 training trims costs in all three seeds","POMO tweak lowers TSP-100 costs, but only for sampled K>1","New training signal beats leader reward, but not at K=1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2248,"prompt_tokens":730,"completion_tokens":1518,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":1446}},"tokens_in":474,"tokens_out":1518,"duration_ms":10348,"temperature":1.0,"reasoning_tokens":1446,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T00:45:25.973154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two arms with six paired seeds under the same frozen protocol; if the stabilized Best-of-8 mean is not lower than leader-reward in at least five of six seeds, the reported direction fails to replicate. A grid over K near 8 and δ near 0.01 would also reveal whether the improvement is a narrow tuned peak or a plateau.","supporting_citations":[],"review_version":1}