{"id":"9b2dea1e-4280-486e-845c-a8cf85fc0b0c","arxiv_id":"2604.08477","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.5,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Curating 25K natural-instruction examples for RLVR yields a 64.4pp relative gain on BBEH for Qwen3-0.6B and generalizes across scales and model families.","lead":"SUPERNOVA is a recipe for turning ordinary human-written instruction data into training fuel for reinforcement learning that improves general LLM reasoning. If the reported gains hold, labs can extend RL-with-verifiable-rewards beyond math and code without inventing new STEM datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Target-aware source-task selection risks evaluation leakage; abstract claims it outperforms average-performance selection without held-out protocols.","rationale":"The reader's weakest_assumption correctly isolates the single most load-bearing premise: that target-aware source selection produces a generalizable RLVR curriculum rather than an overfit one. The abstract's claim that per-target selection beats average-performance selection is exactly the step that would be invalidated by leakage, and no held-out protocol is described. Because the full text and GitHub artifacts are unavailable, the same UNVERDICTED / LOW-confidence posture is appropriate; the concrete test above would settle the issue once the paper and code can be inspected. No stronger internal inconsistency is visible from the abstract alone, and disagreement with consensus is not at issue—this is a methods claim whose validity hinges on clean selection hygiene.","tokens_in":2123,"tokens_out":554,"duration_ms":5777,"concrete_test":"Reconstruct SUPERNOVA using only source tasks chosen by per-target performance on a random half of BBEH tasks (or on a disjoint suite such as BBH/MMLU-Pro); retrain Qwen3-0.6B under identical RLVR settings and evaluate on the held-out BBEH half plus the claimed generalization suites. If the relative gain collapses below ~20pp or fails to transfer, the selection step was leaking evaluation signal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a 64.4pp relative BBEH gain for Qwen3-0.6B after RLVR on SUPERNOVA (25K instances), with generalization to unseen benchmarks, larger scales, and newer families. The abstract states that source-task selection guided by per-target-task performance 'outperforms strategies based on overall average performance' and that this insight guides SUPERNOVA construction. Because BBEH (23 tasks) is both the primary reported target and the selection signal, the load-bearing risk is that the curriculum is overfit to the evaluation suite rather than a general reasoning prior. Without a held-out selection protocol (e.g., selecting sources using only a disjoint subset of BBEH tasks, or using a completely separate validation suite, then measuring the remaining tasks), the large gain and claimed generalization cannot be distinguished from leakage of evaluation signal into the training distribution. Synthetic interventions are reported not to help, which further concentrates the result on this selection step. Abstract-only review cannot inspect absolute scores, baselines, reward construction, or the actual selection procedure, so the concern remains open.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes SUPERNOVA, a framework for curating Reinforcement Learning with Verifiable Rewards (RLVR) data from natural instruction datasets to extend reasoning gains beyond STEM. Through 100+ controlled RL experiments, the authors study three data-design axes—source-task selection, task mixing, and synthetic interventions—and report that selecting source tasks by their effect on individual target tasks outperforms selection by overall average performance, while synthetic interventions do not help. Guided by these findings, they construct a 25K-instance SUPERNOVA dataset and report that RLVR training of Qwen3-0.6B yields a 64.4pp relative gain on BigBench Extra Hard (BBEH; 23 tasks), with claimed generalization to unseen benchmarks, larger model scales, and newer model families. Models, data, and code are released.","tokens_in":2358,"tokens_out":1054,"duration_ms":16649,"significance":"Extending RLVR beyond formal STEM domains is a central open problem; a systematic, empirically grounded recipe for turning existing natural-instruction corpora into verifiable RL data would be practically valuable. The paper’s strengths, as stated, include a large controlled experimental program (100+ runs), an explicit comparison of curation strategies, a concrete 25K dataset, and public release of models, data, and code. If the selection protocol is free of evaluation leakage and the absolute gains and generalization claims hold under standard controls, the work would provide actionable guidance for the community.","major_comments":[{"comment":"The abstract’s central methodological claim is that source-task selection guided by per-target-task performance outperforms selection by overall average performance, and that this insight guides SUPERNOVA construction. BBEH (23 complex reasoning tasks) is both the primary reported target and, on the face of the abstract, the natural selection signal. Without a held-out selection protocol (e.g., selecting sources using only a disjoint subset of BBEH tasks or an entirely separate validation suite, then evaluating the remaining tasks), the 64.4pp relative BBEH gain and the claimed generalization cannot be distinguished from leakage of evaluation signal into the training curriculum. This is load-bearing for the paper’s main result and must be addressed with an explicit selection/evaluation split and ablations against average-performance and random baselines under that split.","section":null},{"comment":"The headline result is reported only as a “relative gain of 64.4pp” of SUPERNOVA-trained Qwen3-0.6B over base Qwen3-0.6B on BBEH. Relative percentage-point gains are hard to interpret without absolute scores for base, SUPERNOVA, and strong alternative RLVR/SFT baselines, plus variance across seeds. The manuscript must report absolute scores, confidence intervals or multi-seed variance, and comparable baselines so that the magnitude of the claim can be assessed and the “relative” wording cannot be misread as absolute points.","section":null},{"comment":"Generalization is claimed to “unseen benchmarks, larger model scales, and newer model families.” Given that source tasks were chosen by per-target performance, the manuscript must define precisely which benchmarks were used in selection versus held out as truly unseen, and show that gains persist on the held-out set. Cross-scale and cross-family transfer should likewise be reported with absolute numbers and the same selection protocol frozen (no re-selection on the larger/newer models), otherwise the generalization claim is not supported.","section":null}],"minor_comments":[{"comment":"Clarify in the abstract (and early text) whether “64.4pp relative gain” means a relative improvement of 64.4% over the base absolute score, or a 64.4 percentage-point absolute difference; the current wording is ambiguous.","section":null},{"comment":"State briefly how verifiable rewards are obtained from natural instruction data (exact-match, LLM judge, programmatic checks, etc.), since reward construction is essential to RLVR and is not specified in the abstract.","section":null},{"comment":"When the full paper is available, ensure tables report absolute scores for all 23 BBEH tasks (or aggregates with per-task breakdowns) so that gains are not driven by a small subset of tasks that were overweighted in selection.","section":null}],"recommendation":"uncertain","confidential_remarks":"This is an abstract-only review; the full manuscript was not available. The load-bearing concern is evaluation leakage from target-aware source-task selection into the reported BBEH numbers. I cannot verify absolute scores, baselines, reward design, or held-out protocols from the abstract alone, so I recommend “uncertain” pending the full paper. If the full paper already contains a clean held-out selection protocol and absolute multi-seed results, the recommendation could move to minor_revision; if selection was done on the same suite later reported, major_revision or reject would be appropriate depending on severity."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is an empirical methods paper on turning natural-instruction corpora into RLVR data for general (non-STEM) reasoning. The new objects are SUPERNOVA (25K curated instances), a controlled study of source-task selection / mixing / synthetic interventions, and a large relative BBEH gain on Qwen3-0.6B with claimed generalization across benches, scale, and families.\n\nWhat it does well, on the abstract alone: the problem is real—RLVR has been stuck on math/code because verifiable rewards are scarce elsewhere—and mining existing human-annotated instruction data is a cheaper path than inventing new formal domains. Running 100+ controlled RL runs on three design axes is the right kind of work. Finding that source-task selection dominates, that per-target selection beats average-performance selection, and that synthetic interventions do not help is a concrete, falsifiable empirical claim. Shipping models, data, and code is real credit; that is how this line of work should be done.\n\nSoft spots, in proportion: we only have the abstract, so absolute scores, baselines, variance, reward construction, and contamination controls are unchecked. The 64.4pp relative gain is uninterpretable without base and absolute numbers. The load-bearing risk is exactly the one the stress note flags: selecting source tasks by their effect on individual target tasks, when BBEH is both the selection signal and the primary reported suite, can bake evaluation signal into the curriculum. The abstract does not describe a held-out selection protocol (e.g., select on a disjoint task subset or a separate validation suite, then measure the rest). That does not kill the paper; it is the first question a referee should force. If they did hold out cleanly, the result is solid methods work. If they did not, the headline gain and “generalization” claims need to be re-read as possible overfit.\n\nWho this is for: people who actually curate RL post-training data and care about non-STEM reasoning. Not a paradigm paper. It deserves a serious referee rather than a desk reject—especially with public artifacts—so the selection protocol and absolute metrics can be inspected. I would not cite from the abstract alone; I would read the full paper and the GitHub before deciding.","headline":"Useful RLVR data-curation study with a real, checkable leakage risk on the headline selection method; abstract-only, but worth full review if the artifacts are real.","tokens_in":2992,"tokens_out":570,"would_cite":false,"duration_ms":14832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Curating 25K natural-instruction examples for RL with verifiable rewards lifts small models by 64pp on hard general-reasoning benchmarks and generalizes across scales.","keywords":["RLVR","natural instructions","source task selection","general reasoning","BigBench Extra Hard","data curation","Qwen3","reinforcement learning"],"falsifier":"Train the same base model on SUPERNOVA after deliberately removing every source task that was selected for any of the 23 BBEH targets and check whether the relative gain on BBEH and held-out suites collapses.","tokens_in":3006,"feed_emoji":"🚀","tokens_out":536,"duration_ms":5168,"temperature":0.7,"pith_summary":"The paper argues that Reinforcement Learning with Verifiable Rewards (RLVR) can move beyond math and code once high-quality verifiable training data is harvested from ordinary natural-instruction collections rather than invented from scratch. After more than a hundred controlled RL runs, the authors find that the decisive design choice is which source tasks to keep: selecting source tasks by how much they help each individual target task beats selection by average performance or by synthetic rewrites. The resulting 25K-instance SUPERNOVA set, trained on Qwen3-0.6B, produces a 64.4 percentage-point relative gain on BigBench Extra Hard and the same gains transfer to held-out benchmarks, larger models, and newer model families. A sympathetic reader cares because this supplies a concrete, reproducible recipe for turning abundant human-annotated instructions into general reasoning fuel without needing new STEM-style verifiers.","feed_headline":"25K natural-instruction RL examples lift hard reasoning 64pp","feed_subtitle":"Gains hold on unseen tests, larger models, and newer families without STEM verifiers","key_machinery":"Source-task selection guided by per-target-task performance: for each evaluation task, retain only those natural-instruction sources that improve that task under RLVR, then mix the retained sources into a single 25K-instance training set.","core_discovery":"Training a 0.6B model on SUPERNOVA—a 25K-instance RLVR set curated from natural instruction data by selecting source tasks according to their per-target-task performance—yields a 64.4 percentage-point relative gain on BigBench Extra Hard and generalizes to unseen benchmarks, larger scales, and newer model families.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["25K natural-instruction RL set lifts BBEH 64pp via task selection","SUPERNOVA: per-target curation of natural data yields 64pp reasoning gain","Source-task selection on human instructions boosts hard reasoning 64pp","25K RLVR instances from natural tasks improve BBEH 64pp on 0.6B","Natural-instruction curation drives 64pp BBEH gain that generalizes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That picking source tasks by how well they help the chosen evaluation suite produces a generalizable curriculum rather than simply overfitting those same targets.","fun_headline_variants_meta":{"raw":{"variants":["25K natural-instruction RL set lifts BBEH 64pp via task selection","SUPERNOVA: per-target curation of natural data yields 64pp reasoning gain","Source-task selection on human instructions boosts hard reasoning 64pp","25K RLVR instances from natural tasks improve BBEH 64pp on 0.6B","Natural-instruction curation drives 64pp BBEH gain that generalizes"]},"model":"grok-4.5","effort":"low","cost_usd":0.005238,"raw_usage":{"total_tokens":1521,"prompt_tokens":876,"num_sources_used":0,"completion_tokens":93,"cost_in_usd_ticks":52380000,"prompt_tokens_details":{"text_tokens":876,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":552,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":876,"tokens_out":93,"duration_ms":4848,"temperature":1.0,"reasoning_tokens":552,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T23:50:01.784475+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same base model on SUPERNOVA after deliberately removing every source task that was selected for any of the 23 BBEH targets and check whether the relative gain on BBEH and held-out suites collapses.","supporting_citations":[],"review_version":2}