SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions

Ashima Suvarna; Hritik Bansal; Kendrick Phan; Mehrab Beikzadeh; Saadia Gabriel

Curating 25K natural-instruction examples for RL with verifiable rewards lifts small models by 64pp on hard general-reasoning benchmarks and generalizes across scales.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 23:50 UTC pith:2GWQW6DN

load-bearing objection Useful RLVR data-curation study with a real, checkable leakage risk on the headline selection method; abstract-only, but worth full review if the artifacts are real. the 3 major comments →

arxiv 2604.08477 v2 pith:2GWQW6DN submitted 2026-04-09 cs.AI cs.CLcs.LG

SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions

Ashima Suvarna , Kendrick Phan , Mehrab Beikzadeh , Hritik Bansal , Saadia Gabriel This is my paper

classification cs.AI cs.CLcs.LG

keywords RLVRnatural instructionssource task selectiongeneral reasoningBigBench Extra Harddata curationQwen3reinforcement learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that Reinforcement Learning with Verifiable Rewards (RLVR) can move beyond math and code once high-quality verifiable training data is harvested from ordinary natural-instruction collections rather than invented from scratch. After more than a hundred controlled RL runs, the authors find that the decisive design choice is which source tasks to keep: selecting source tasks by how much they help each individual target task beats selection by average performance or by synthetic rewrites. The resulting 25K-instance SUPERNOVA set, trained on Qwen3-0.6B, produces a 64.4 percentage-point relative gain on BigBench Extra Hard and the same gains transfer to held-out benchmarks, larger models, and newer model families. A sympathetic reader cares because this supplies a concrete, reproducible recipe for turning abundant human-annotated instructions into general reasoning fuel without needing new STEM-style verifiers.

Core claim

Training a 0.6B model on SUPERNOVA—a 25K-instance RLVR set curated from natural instruction data by selecting source tasks according to their per-target-task performance—yields a 64.4 percentage-point relative gain on BigBench Extra Hard and generalizes to unseen benchmarks, larger scales, and newer model families.

What carries the argument

Source-task selection guided by per-target-task performance: for each evaluation task, retain only those natural-instruction sources that improve that task under RLVR, then mix the retained sources into a single 25K-instance training set.

Load-bearing premise

That picking source tasks by how well they help the chosen evaluation suite produces a generalizable curriculum rather than simply overfitting those same targets.

What would settle it

Train the same base model on SUPERNOVA after deliberately removing every source task that was selected for any of the 23 BBEH targets and check whether the relative gain on BBEH and held-out suites collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful RLVR data-curation study with a real, checkable leakage risk on the headline selection method; abstract-only, but worth full review if the artifacts are real.

read the letter

Punchline: this is an empirical methods paper on turning natural-instruction corpora into RLVR data for general (non-STEM) reasoning. The new objects are SUPERNOVA (25K curated instances), a controlled study of source-task selection / mixing / synthetic interventions, and a large relative BBEH gain on Qwen3-0.6B with claimed generalization across benches, scale, and families.

What it does well, on the abstract alone: the problem is real—RLVR has been stuck on math/code because verifiable rewards are scarce elsewhere—and mining existing human-annotated instruction data is a cheaper path than inventing new formal domains. Running 100+ controlled RL runs on three design axes is the right kind of work. Finding that source-task selection dominates, that per-target selection beats average-performance selection, and that synthetic interventions do not help is a concrete, falsifiable empirical claim. Shipping models, data, and code is real credit; that is how this line of work should be done.

Soft spots, in proportion: we only have the abstract, so absolute scores, baselines, variance, reward construction, and contamination controls are unchecked. The 64.4pp relative gain is uninterpretable without base and absolute numbers. The load-bearing risk is exactly the one the stress note flags: selecting source tasks by their effect on individual target tasks, when BBEH is both the selection signal and the primary reported suite, can bake evaluation signal into the curriculum. The abstract does not describe a held-out selection protocol (e.g., select on a disjoint task subset or a separate validation suite, then measure the rest). That does not kill the paper; it is the first question a referee should force. If they did hold out cleanly, the result is solid methods work. If they did not, the headline gain and “generalization” claims need to be re-read as possible overfit.

Who this is for: people who actually curate RL post-training data and care about non-STEM reasoning. Not a paradigm paper. It deserves a serious referee rather than a desk reject—especially with public artifacts—so the selection protocol and absolute metrics can be inspected. I would not cite from the abstract alone; I would read the full paper and the GitHub before deciding.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes SUPERNOVA, a framework for curating Reinforcement Learning with Verifiable Rewards (RLVR) data from natural instruction datasets to extend reasoning gains beyond STEM. Through 100+ controlled RL experiments, the authors study three data-design axes—source-task selection, task mixing, and synthetic interventions—and report that selecting source tasks by their effect on individual target tasks outperforms selection by overall average performance, while synthetic interventions do not help. Guided by these findings, they construct a 25K-instance SUPERNOVA dataset and report that RLVR training of Qwen3-0.6B yields a 64.4pp relative gain on BigBench Extra Hard (BBEH; 23 tasks), with claimed generalization to unseen benchmarks, larger model scales, and newer model families. Models, data, and code are released.

Significance. Extending RLVR beyond formal STEM domains is a central open problem; a systematic, empirically grounded recipe for turning existing natural-instruction corpora into verifiable RL data would be practically valuable. The paper’s strengths, as stated, include a large controlled experimental program (100+ runs), an explicit comparison of curation strategies, a concrete 25K dataset, and public release of models, data, and code. If the selection protocol is free of evaluation leakage and the absolute gains and generalization claims hold under standard controls, the work would provide actionable guidance for the community.

major comments (3)

The abstract’s central methodological claim is that source-task selection guided by per-target-task performance outperforms selection by overall average performance, and that this insight guides SUPERNOVA construction. BBEH (23 complex reasoning tasks) is both the primary reported target and, on the face of the abstract, the natural selection signal. Without a held-out selection protocol (e.g., selecting sources using only a disjoint subset of BBEH tasks or an entirely separate validation suite, then evaluating the remaining tasks), the 64.4pp relative BBEH gain and the claimed generalization cannot be distinguished from leakage of evaluation signal into the training curriculum. This is load-bearing for the paper’s main result and must be addressed with an explicit selection/evaluation split and ablations against average-performance and random baselines under that split.
The headline result is reported only as a “relative gain of 64.4pp” of SUPERNOVA-trained Qwen3-0.6B over base Qwen3-0.6B on BBEH. Relative percentage-point gains are hard to interpret without absolute scores for base, SUPERNOVA, and strong alternative RLVR/SFT baselines, plus variance across seeds. The manuscript must report absolute scores, confidence intervals or multi-seed variance, and comparable baselines so that the magnitude of the claim can be assessed and the “relative” wording cannot be misread as absolute points.
Generalization is claimed to “unseen benchmarks, larger model scales, and newer model families.” Given that source tasks were chosen by per-target performance, the manuscript must define precisely which benchmarks were used in selection versus held out as truly unseen, and show that gains persist on the held-out set. Cross-scale and cross-family transfer should likewise be reported with absolute numbers and the same selection protocol frozen (no re-selection on the larger/newer models), otherwise the generalization claim is not supported.

minor comments (3)

Clarify in the abstract (and early text) whether “64.4pp relative gain” means a relative improvement of 64.4% over the base absolute score, or a 64.4 percentage-point absolute difference; the current wording is ambiguous.
State briefly how verifiable rewards are obtained from natural instruction data (exact-match, LLM judge, programmatic checks, etc.), since reward construction is essential to RLVR and is not specified in the abstract.
When the full paper is available, ensure tables report absolute scores for all 23 BBEH tasks (or aggregates with per-task breakdowns) so that gains are not driven by a small subset of tasks that were overweighted in selection.

Circularity Check

0 steps flagged

Empirical RLVR curation paper; mild risk that target-aware source selection bakes evaluation signal into the curriculum, but no definitional tautology or forced-by-construction prediction.

full rationale

SUPERNOVA is an empirical data-curation and RLVR training paper, not a first-principles derivation. Its central claims are experimental outcomes (64.4pp relative BBEH gain for Qwen3-0.6B; claimed generalization to unseen benchmarks, larger scales, and newer families). There are no equations that redefine the reported metric as an input, no uniqueness theorems imported from the authors, no ansatz smuggled via self-citation, and no renaming of a known closed-form result. The only load-bearing methodological choice visible in the abstract is source-task selection guided by per-target-task performance, which can in principle leak evaluation signal into the training mix if the same BBEH tasks used for selection are later reported as primary results. That is a validity/leakage concern rather than circularity by construction: the abstract also asserts that gains transfer to unseen benchmarks, which—if true under proper held-out protocols—breaks pure circularity. Abstract-only review cannot inspect selection folds, absolute scores, or reward design, so no specific reduction (Eq. X ≡ Eq. Y, or fitted parameter renamed as prediction) can be exhibited. Per the analyzer rules, residual methodological risk without a quotable by-construction collapse warrants a low score (0–2), not 6+. Score 2 reflects that mild residual concern without manufacturing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Abstract-only audit. The work rests on standard RLVR assumptions plus the domain claim that natural instruction datasets supply usable verifiable rewards. No free parameters are numerically fitted in the abstract; the main design choice is empirical source-task selection. No new physical or mathematical entities are invented.

axioms (3)

domain assumption Natural instruction datasets contain (or can be filtered into) instances with verifiable rewards suitable for RLVR outside formal STEM domains.
Central premise of the SUPERNOVA framework; without verifiable rewards, RLVR does not apply.
ad hoc to paper Source-task selection based on per-target performance is a valid curriculum design that transfers beyond the selection targets.
Abstract reports this strategy outperforms average-performance selection; transfer is assumed for generalization claims.
domain assumption Standard RLVR training dynamics and reward-model/verifier assumptions used in math/code RLVR carry over to natural-instruction data.
Background of the entire experimental program.

pith-pipeline@v1.1.0-grok45 · 6246 in / 2292 out tokens · 27219 ms · 2026-07-12T23:50:01.784475+00:00 · methodology

0 comments

read the original abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging. Extending RLVR beyond STEM is fundamentally constrained by the lack of high-quality verifiable training data. In this work, we introduce SUPERNOVA, a framework for curating RLVR data from natural instruction datasets, which are a rich source of expert-annotated data but are underexplored for RLVR training. Through 100+ controlled RL experiments, we systematically study how to utilize these dataset for RLVR and how data curation decisions affect downstream reasoning performance . In particular, we investigate three data designs: (a) source task selection, (b) task mixing, and (c) synthetic interventions. Our analysis reveals that source task selection has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance and synthetic interventions do not improve reasoning. Guided by these insights, we construct SUPERNOVA, a high-quality RLVR dataset of 25K instances curated from natural instruction datasets. We show that training Qwen3-0.6B on SUPERNOVA outperforms the base Qwen3-0.6B, yielding a relative gain of 64.4pp on BigBench Extra Hard (BBEH), a challenging benchmark comprising 23 complex reasoning tasks. Importantly, we find that gains from SUPERNOVA generalize to unseen benchmarks, larger model scales, and newer model families. Overall, our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Models, Data, Code at https://github.com/asuvarna31/supernova.