{"id":"98aa3454-5841-48bb-b84d-314e0147d5ed","arxiv_id":"2602.17684","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CodeScaler reward model outperforms execution-based RL on code benchmarks by up to 4.23 points, scales training to 44K problems without test cases for +14.64 point gains, and offers 10x lower latency at inference.","lead":"CodeScaler is a reward model trained on preference data from verified code problems to improve reinforcement learning for code generation in LLMs, both during training and at inference time. It aims to reduce reliance on unit tests, enabling larger scale training and faster inference while reporting gains on coding benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Reward model trained on curated verified problems may not reliably proxy execution correctness on synthetic data or new problems, undermining no-test-case scaling claims.","rationale":"The reader's weakest assumption matches the load-bearing point exactly. Full-text details on preference construction, RM accuracy ablations against execution, and OOD evaluation would be needed to resolve it; absent those, the no-test-case scaling remains conditional on RM fidelity. This is an internal consistency risk rather than external consensus disagreement.","tokens_in":1816,"tokens_out":356,"duration_ms":34412,"concrete_test":"Take 200 held-out problems with fresh unit tests (post any training cutoff), sample 8 generations each from the base model, obtain binary correctness labels via execution, and compute (a) Spearman rank correlation between RM scores and execution pass rates and (b) top-1 selection accuracy of RM vs. execution; if correlation <0.75 or selection gap >8 points, the generalization assumption fails and scaling claims weaken.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central scaling results (+14.64 points on 44K synthetic problems without any test cases, plus test-time latency reduction) require CodeScaler to act as a faithful surrogate for unit-test rewards during both RL training and inference-time selection. Preference data is constructed from 'verified code problems' (implying execution-based labeling to create positive/negative pairs), then augmented with synthetic data and shaped via syntax-aware extraction. If the RM captures dataset-specific artifacts rather than semantic correctness, optimization will exploit RM errors and gains will not transfer. The reported +3.3 on RM-Bench and cross-domain gains do not directly measure correlation with actual execution outcomes on the target benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces CodeScaler, a reward model trained on curated preference data from verified code problems using syntax-aware code extraction and validity-preserving reward shaping. It claims to outperform execution-based RL on four coding benchmarks (+1.55 points on Qwen3-8B-Base and +4.23 on Qwen3-14B-Base), enable scaling RL training to 44K synthetic problems for +14.64 points over the base model without any test cases, serve as an effective test-time scaling method with performance comparable to unit tests but 10x lower latency, and outperform prior reward models on RM-Bench in code (+3.3) and other domains (+2.7 average).","tokens_in":1967,"tokens_out":571,"duration_ms":30698,"significance":"If the results hold and the reward model reliably proxies execution correctness, CodeScaler would meaningfully advance scalable code LLM training by removing the unit-test bottleneck and enabling larger synthetic datasets. The reported latency reduction at inference and cross-domain RM-Bench gains would be practically useful. The design choices around syntax-aware extraction and reward shaping are sensible for robustness and deserve credit as thoughtful engineering contributions.","major_comments":[{"comment":"The central scaling claim (+14.64 points on 44K synthetic problems without test cases) is load-bearing for the no-test-case contribution, yet the manuscript provides no direct correlation analysis between CodeScaler reward scores and actual execution outcomes on those synthetic problems. Without this, it remains possible that gains arise from RM exploitation of curation artifacts rather than semantic generalization (see scaling experiments section).","section":"Scaling Experiments"},{"comment":"The preference data construction from verified code problems (used to train the RM) is described at a high level but lacks explicit details on pair creation, data exclusion rules, and any post-hoc filtering. This information is necessary to evaluate whether the reported outperformance over execution-based RL could partly reflect choices in the training distribution rather than independent generalization.","section":"Method"}],"minor_comments":[{"comment":"The abstract refers to 'four coding benchmarks' without naming them; listing the specific benchmarks (e.g., HumanEval, MBPP) would improve clarity for readers.","section":"Abstract"},{"comment":"Reported numeric gains lack error bars, number of runs, or statistical significance tests; adding these would strengthen the experimental claims.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":"The reader's low soundness score and the skeptic's concern about RM proxy reliability are well-founded given the missing validation details; the manuscript would benefit from additional experiments correlating RM scores with execution before the scaling claims can be fully trusted."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation of our results and methods.","responses":[{"response":"We appreciate the referee's emphasis on validating the reward model's behavior on the scaled synthetic data. Although the 44K problems are constructed without test cases to highlight the removal of the execution bottleneck, we agree that a direct correlation analysis would further support the claim of semantic generalization. In the revised manuscript, we will include such an analysis on a held-out subset of problems where execution outcomes can be obtained or generated, reporting the correlation between CodeScaler scores and pass rates to address concerns about potential curation artifacts.","revision_made":"yes","referee_comment":"[Scaling Experiments] The central scaling claim (+14.64 points on 44K synthetic problems without test cases) is load-bearing for the no-test-case contribution, yet the manuscript provides no direct correlation analysis between CodeScaler reward scores and actual execution outcomes on those synthetic problems. Without this, it remains possible that gains arise from RM exploitation of curation artifacts rather than semantic generalization (see scaling experiments section)."},{"response":"We agree that additional methodological details would enhance reproducibility and allow readers to better assess the training distribution. In the revised manuscript, we will expand the preference data construction section to provide explicit descriptions of pair creation from verified problems, the data exclusion rules employed, and any post-hoc filtering steps applied to maintain data quality and robustness.","revision_made":"yes","referee_comment":"[Method] The preference data construction from verified code problems (used to train the RM) is described at a high level but lacks explicit details on pair creation, data exclusion rules, and any post-hoc filtering. This information is necessary to evaluate whether the reported outperformance over execution-based RL could partly reflect choices in the training distribution rather than independent generalization."}],"tokens_in":1444,"tokens_out":415,"duration_ms":30166,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"CodeScaler trains a reward model on preference pairs from verified code problems and uses it to drive RL training plus test-time selection for code LLMs. The main reported wins are beating execution-based RL by a few points on Qwen3 models and then jumping +14.64 points when scaling to 44K synthetic problems with no test cases at all. At inference it matches unit-test performance while cutting latency by roughly 10x, and it also edges out other reward models on RM-Bench in code and other domains.","headline":"CodeScaler replaces unit tests with a learned reward model for scaling code RL training and inference, but the generalization from verified to synthetic data looks like the part that needs checking.","tokens_in":2481,"tokens_out":184,"would_cite":false,"duration_ms":23232,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"CodeScaler reward-model pipeline (Bradley-Terry + syntax shaping) has no structural overlap with RS cost or forcing chain","alignment":"orthogonal","rationale":"The paper's central machinery is empirical preference-data curation, AST-based extraction, validity-preserving shaping of a learned scalar reward, and GRPO optimization on synthetic code problems. None of these components invoke, parallel, or contradict the RS derivation of J(x) = ½(x + x⁻¹) − 1, the golden-ratio ladder, 8-tick periodicity, or parameter-free emergence of c, ℏ, G. The work lies entirely in the domain of LLM post-training and is therefore orthogonal to the RS framework.","tokens_in":62045,"confidence":"high","tokens_out":162,"duration_ms":10774,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A reward model trained on verified code preferences scales both RL training and test-time inference for code LLMs without needing unit tests.","keywords":["code generation","reward models","reinforcement learning","large language models","test-time scaling","code LLMs","preference data"],"falsifier":"Apply the trained CodeScaler reward model to a fresh coding benchmark outside the training distribution and measure whether performance gains disappear relative to execution-based RL.","tokens_in":2714,"feed_emoji":"📈","tokens_out":453,"duration_ms":21982,"temperature":0.7,"pith_summary":"The paper proposes CodeScaler to overcome the dependence of code LLMs on scarce or unreliable unit tests for reinforcement learning. It trains a reward model on carefully curated preference data from verified problems, using syntax-aware extraction and validity-preserving reward shaping. This produces consistent gains over execution-based RL across benchmarks and allows scaling training data to 44K problems with synthetic data alone. At inference, the same model matches unit-test performance while cutting latency by a factor of ten and also improves on a general reward-model benchmark.","feed_headline":"Reward model scales code LLM training without unit tests","feed_subtitle":"CodeScaler trained on verified preferences beats execution RL on benchmarks and cuts inference latency tenfold.","key_machinery":"CodeScaler reward model, trained with syntax-aware code extraction and validity-preserving reward shaping on preference data from verified code problems.","core_discovery":"CodeScaler is a reward model trained on preference data derived from verified code problems that incorporates syntax-aware code extraction and validity-preserving reward shaping; when used for RL training it outperforms execution-based methods by 1.55 points on Qwen3-8B-Base and 4.23 points on Qwen3-14B-Base, yields a 14.64-point gain over the base model when scaled to 44K problems without any test cases, and at inference time delivers unit-test-comparable accuracy with a ten-fold latency reduction.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["CodeScaler scales code LLM training with reward models","CodeScaler beats execution RL on four coding benchmarks","CodeScaler matches unit tests with tenfold latency reduction","CodeScaler delivers 14 point gains without test cases"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Preference data derived from verified code problems produces a reward model that generalizes reliably to new problems without overfitting or systematic bias.","fun_headline_variants_meta":{"raw":{"variants":["CodeScaler scales code LLM training with reward models","CodeScaler beats execution RL on four coding benchmarks","CodeScaler matches unit tests with tenfold latency reduction","CodeScaler delivers 14 point gains without test cases"]},"model":"grok-4.3","cost_usd":0.008737,"raw_usage":{"total_tokens":3871,"prompt_tokens":699,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":87365500,"prompt_tokens_details":{"text_tokens":699,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3113,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":699,"tokens_out":59,"duration_ms":42788,"temperature":1.0,"reasoning_tokens":3113,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-21T13:28:06.194886+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Apply the trained CodeScaler reward model to a fresh coding benchmark outside the training distribution and measure whether performance gains disappear relative to execution-based RL.","supporting_citations":[],"review_version":1}