{"id":"8cf85f66-d209-4aac-b468-8edd599ca2ee","arxiv_id":"2505.14147","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A self-alignment prompt pipeline (SHARP) generated roughly 190,000 verifiable STEM problems that, when used to distill or RL-train 7B models, improved GPQA Diamond accuracy in the reported runs.","lead":"SHARP is a pipeline that uses a large reasoning model to generate graduate-level STEM problems with verifiable numeric answers, then trains other models on those problems. The authors report that this synthetic data improves GPQA benchmark scores for a 7B model, though the gains come without error bars or released data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'verified ground truth' premise is not yet supported: the chemistry example in Table 4 appears stoichiometrically inconsistent, so a non-trivial fraction of SHARP reference answers could be wrong and corrupt both distillation labels and RLVR rewards.","rationale":"The reader's weakest assumption is exactly the point I would stress: the pipeline relies on model-generated reference answers as ground truth without independent validation. I agree with the reader's conditional verdict. My addition is a concrete demonstration from the paper itself: the Table 4 chemistry problem appears internally inconsistent under standard stoichiometry, which shows that the verification stage can pass a flawed reference answer. This makes the concern more than a theoretical risk and strengthens the case that the authors should release the dataset and provide an expert audit before the claim is accepted. I also considered the benchmark-alignment issue, since the Seed Topics Library is deliberately sampled to match GPQA knowledge-point distributions and the paper evaluates only on GPQA. That is a real limitation for generalization claims, but it is secondary to the correctness of the training signal: even a GPQA-aligned dataset would be valid if the labels were correct. The correctness of the reference answers is the weakest structural link, so I recommend keeping the reader's CONDITIONAL verdict unchanged while making the audit an explicit condition.","tokens_in":26638,"tokens_out":7869,"duration_ms":77520,"concrete_test":"Have two domain experts independently solve a random sample of 100 SHARP-generated problems (including the Table 4 chemistry item as a control) and compare their answers against the stored reference answers. If the experts disagree with more than 5% of reference answers, or if the Table 4 chemistry item is judged unsolvable as stated, the ground-truth premise fails and the reported improvements cannot be cleanly attributed to correct labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SHARP's central claim depends on the correctness of the generated reference answers. During distillation these answers are used as supervision labels, and during RL Zero the reward in Appendix B.2 is R_acc(a_i, a'_i), which awards 1 iff the model's answer 'is similar to the ground truth solution a'_i'. But a'_i is itself produced by the same LRM pipeline and filtered only by a model self-check plus Math-Verify, which checks format and string equality, not domain-level correctness. No human expert validation is reported. This is structurally load-bearing: if any meaningful fraction of reference answers is wrong, RLVR rewards and distillation labels systematically reinforce incorrect reasoning, and the reported GPQA gains cannot be attributed to learning correct science. The concern is not merely hypothetical. The chemistry problem in Table 4 states that CO2 mass loss is 2.64 g, which implies 0.0600 mol ZnCO3 and therefore 3.92 g of zinc, yet it also states that 5.89 g of zinc metal is produced, which requires 0.0901 mol of ZnO. The stated reference answer of 58.9% is not derivable from the given data without an unstated additional assumption. This example passed the SHARP verification pipeline, demonstrating that the quality control is not airtight. A single flawed item does not prove dataset-wide failure, but it shifts the burden: the paper's 'verified ground truth' language is stronger than the evidence supports, and the conditional acceptance should explicitly require an expert audit.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SHARP, a three-phase pipeline (Alignment, Instantiation, Inference) plus a Verifying stage for synthesizing graduate- and Olympiad-level STEM reasoning problems with a state-of-the-art LRM (DeepSeek R1). The pipeline is governed by explicit self-alignment principles, uses a three-tier subject taxonomy seeded by a topic library that is partly derived from GPQA knowledge-point distributions, and filters outputs through a model self-check and Math-Verify. The resulting problems are used for SFT distillation and for GRPO-based RLVR training. Experiments report GPQA Diamond accuracy of 54.7 for a SHARP-distilled Qwen2.5-7B model versus 46.4 for the baseline and 49.9 for DeepSeek-R1-Distill-Qwen-7B, and 37.0 versus 35.5 for RL-Zero training. The paper claims that SHARP-augmented training substantially improves complex STEM reasoning.","tokens_in":26928,"tokens_out":6663,"duration_ms":67390,"significance":"If the claims hold, SHARP would provide a scalable route to generating verifiable hard STEM training data with reduced human annotation, which is a valuable contribution. The paper is unusually detailed about its pipeline, prompts, hyperparameters, and dataset-distribution analyses, and the distillation gains (+8.3 points on GPQA Diamond) are substantial and consistent across subjects. However, two load-bearing issues temper the significance. First, the training-data topic distribution is deliberately aligned to the evaluation benchmark, so the GPQA results are partly a benchmark-loop artifact. Second, the 'verified ground truth' is model-generated and filtered without human validation; the chemistry example in Table 4 is internally inconsistent, so the verification stage demonstrably passes domain-level errors. The RL-Zero gain of 1.5 points is also within plausible sampling noise. These issues must be addressed before the general claim of expert-level STEM reasoning gains is established.","major_comments":[{"comment":"The seed-topic library is constructed in part by extracting topic keypoints from GPQA, and §C.1 states that the authors 'sampled according to the distribution of disciplines and the corresponding knowledge points' so that the synthetic data 'fully covers the relevant knowledge points of GPQA.' Evaluation is then reported on GPQA. The authors state that they do not directly rephrase validation queries, and this is a meaningful distinction, but aligning the training-data topic distribution to the test benchmark is still a benchmark-loop burden: the reported gains may reflect distribution matching rather than improved general reasoning. To support the paper's central claim, the authors should evaluate on held-out STEM benchmarks (e.g., MMLU-STEM, SciBench, or other graduate-level STEM sets) or, at minimum, ablate the evaluation with topic sampling that is independent of GPQA. The paper's decontamination step is described as exact-duplicate removal; it does not address this topical overlap.","section":"§3.2, §C.1, Tables 1–2"},{"comment":"The reference answers used for both distillation labels and RLVR rewards are generated by the same LRM pipeline and filtered only by a model self-check and Math-Verify. Appendix B.2 defines R_acc(a_i, a'_i) = 1 iff the model's answer 'is similar to the ground truth solution a'_i,' but a'_i is itself the pipeline's output. No human expert validation is reported. The chemistry example in Table 4 shows that this verification is not airtight: a mass loss of 2.64 g of CO2 implies 0.0600 mol of ZnCO3 and therefore 3.92 g of zinc, whereas the problem states that 5.89 g of zinc is produced, which requires 0.0901 mol of ZnO and a corresponding CO2 loss of 3.96 g. The stated reference answer of 58.9% is not derivable from the data as written without an unstated additional assumption. This example passed the SHARP verification pipeline, demonstrating that domain-level correctness is not reliably enforced. Since incorrect reference answers directly corrupt the reward signal and the distillation supervision, the paper should quantify reference-answer accuracy (e.g., a human expert audit of a random sample, or agreement with an independent solver) and either strengthen the verification or soften the 'verified ground truth' language.","section":"§3.3, Appendix B.2, Table 4, Algorithm 1"},{"comment":"The RL-Zero result in Table 2 reports a GPQA Diamond gain of 1.5 points (37.0 vs. 35.5). GPQA Diamond contains roughly 198 questions, so this difference corresponds to about 3 questions and lies within one binomial standard error (approximately 3.4 points). The paper's own NeurIPS checklist item 7 answers 'No' to error bars or statistical significance tests. Appendix B.2 states that accuracy is averaged over '16 independent inference runs,' but no variance or confidence interval is reported, and it is unclear what varies between runs if decoding is greedy. The abstract's claim that SHARP-augmented training 'substantially outperforms existing methods' is not supported by the RL-Zero evidence alone. Please report confidence intervals, the number of evaluation questions, and the run-to-run variance; if the 16 runs are independent samples, report the standard error across them.","section":"Table 2, Checklist item 7"}],"minor_comments":[{"comment":"The first example in Table 4 is labeled 'Particle Physics, High Energy Physics' but the problem text opens with 'Solve the following chemical problem'; this subject-label/content mismatch should be corrected.","section":"Table 4"},{"comment":"The Verifying stage is described only as checking that 'all alignment checks' pass; the concrete checks performed by the model self-check and by Math-Verify, and any thresholds used (e.g., confidence score, pass-rate filter), are not specified. Please provide these details, including the pass-rate filtering threshold used to reduce the 229,452 generated pairs to the reported 190,000.","section":"Algorithm 1, §3.3 (Verifying)"},{"comment":"The reward function R_acc uses 'equal(a_i, a'_i)' but the equality relation is not defined; please specify whether it is exact string matching, numeric tolerance, or Math-Verify equivalence.","section":"Appendix B.2"},{"comment":"The pass-rate comparisons use different numbers of responses for the two models (10 vs. 5 for physics, 5 vs. 1 for biology, 10 vs. 5 for chemistry in the text and figures), which makes the cross-model comparisons difficult to interpret; please standardize the evaluation protocol or justify the differences.","section":"Appendix C.3"},{"comment":"The dataset size is reported as 190,000 in Section 4 and as 229,452 generated in Appendix C; while the appendix clarifies that about 190,000 remain after filtering, the main text should state both the generated and post-filter counts for consistency. Also, the phrase 'disinfection' in Appendix C.2 should be 'detoxification' (as used elsewhere) or be defined.","section":"Section 4, Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern that the Table 4 chemistry example is stoichiometrically inconsistent; this is not a mere typo, because it passed the paper's own verification pipeline and therefore bears directly on the reliability of the training signal. The benchmark-loop issue is also real, though it is an evaluation-design problem rather than a fraud concern. The distillation results are strong enough that I would not reject the paper, but the authors should be asked for out-of-distribution evaluation, a human audit of reference answers, and proper uncertainty reporting. The paper's current abstract overstates the strength of the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The SHARP pipeline is a real, fairly complete recipe for generating hard STEM problems at scale, and the paper's own Table 4 contains a chemistry problem that is internally inconsistent, which is a direct hit on the 'verified ground truth' claim.\n\nThe genuinely new pieces are the explicit self-alignment constraint set and the three-tier category seed library. The empirical comparison on GPQA is new, and the distillation result is striking: SHARP-Qwen2.5-7B-Instruct-Distill reaches 54.7 on GPQA Diamond versus 46.4 for the baseline and 49.9 for DeepSeek-R1-Distill-Qwen-7B. The paper also includes ablations across physics/chemistry/biology and pass-rate difficulty analyses, which is more than many synthetic-data papers do.\n\nThe soft spots are real. First, the stress-test note holds up. The zinc carbonate problem in Table 4 states a CO2 mass loss of 2.64 g, which implies 0.0600 mol ZnCO3 and 3.92 g of zinc, yet it also states 5.89 g of zinc metal is produced, requiring 0.0901 mol ZnO. The stated answer of 58.9% is not derivable from the given data. This example passed the SHARP verification pipeline, which combines a model self-check and Math-Verify. That verifier checks format and string equality, not domain-level correctness. One flawed item doesn't prove dataset-wide failure, but it shifts the burden: the 'verified ground truth' language is stronger than the evidence supports. The RL reward and distillation labels depend on these reference answers being correct, so this is load-bearing. Second, the paper's own checklist admits no error bars or statistical tests (item 7: No). The RL gain of 1.5 points on GPQA Diamond is within plausible sampling noise for a roughly 198-question benchmark. Third, the training data is deliberately sampled to match the GPQA knowledge-point distribution, so the benchmark-loop burden is real, even if the paper does not copy GPQA questions. Fourth, code and data are not released; the checklist says they will be open-sourced during review, but that is not the same as having them.\n\nWho is this for? Researchers working on synthetic data for reasoning models, especially RLVR. The paper is a legitimate engineering contribution and deserves a serious referee. The referee should require an expert audit of a sample of generated problems, variance estimates or multiple seeds, and at least one held-out benchmark whose topic distribution was not used to seed SHARP.","headline":"A plausible and useful synthetic-data pipeline for STEM reasoning, undercut by an unverified 'ground-truth' premise and thin statistics; the chemistry example in Table 4 is genuinely inconsistent.","tokens_in":27502,"tokens_out":2667,"would_cite":false,"duration_ms":45707,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:39:13.876155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}