{"id":"30e7dd5c-751b-44d4-be15-3ff2bb26b31c","arxiv_id":"2505.23754","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A large web-mined dataset plus variant-consistency reinforcement learning lifts a 7B model to 47% average accuracy on olympiad-style theorem benchmarks, beating similar open models but not top commercial ones.","lead":"DeepTheorem is a 121,000-theorem dataset of informal math statements with AI-written proofs, paired with a reinforcement learning method that trains language models to judge whether each theorem is true or false. It matters because the variant-based reward trick could generalize to other open-ended reasoning tasks where a formal verifier is unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Variant ground-truth labels are the load-bearing risk: LLM-generated 'contradictory' variants may not be logically false, yet both the RL reward and the outcome metric in Fig. 6 depend on those labels.","rationale":"The reader's CONDITIONAL verdict is appropriate. My stress-test identifies the same load-bearing point as the reader's weakest_assumption, sharpened to the variant-label step because it corrupts both training and evaluation. The concern is not about author intent; the pipeline is transparent and the dataset and code are released, which is genuine credit. The issue is epistemic: every downstream number inherits the unverified variant labels. The paper's own Table 2 and the Section 4.3 case study are internal evidence that the LLM-only loop can be too lenient. The SOTA wording in the abstract is also overstated relative to Table 5 (o3-mini 68.28 vs 47.22), but that is a claim-calibration issue, not a correctness flaw. A human audit of variant labels is the decisive missing experiment; if it passes, the results would be much stronger. Hence CONDITIONAL stands.","tokens_in":25290,"tokens_out":6021,"duration_ms":59885,"concrete_test":"Select 100 original theorems from DeepTheorem with their generated variants. Have two human experts independently judge, for each entailing and contradictory variant, whether the variant's label (true/false) follows from the original theorem's hypotheses. Measure inter-annotator agreement and error rate against the LLM labels. If the error rate exceeds 5%, recompute Table 4 outcome scores and the RL reward accuracy using only variants that pass the human audit; if the DeepTheorem-vs-OpenR1 margin and RL-vs-SFT margin change materially, the reported improvements are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the correctness of the variant labels produced in Section 3.1. The variant-construction prompt (Appendix A.8) asks an LLM to negate the original statement, but for theorems of the form ∀x (H(x) → C(x)), negating only the conclusion yields ∀x (H(x) → ¬C(x)), which is not the logical negation and can be vacuously true wherever H fails. The paper's own motivating example (Table 2) shows 'x>1' with 'contradictory variant' 'x<1', which is not contradictory unless the hypotheses explicitly exclude x=1. The pipeline's 'completeness annotation' is another LLM pass with no reported error rate or human audit. This matters twice over: the RL reward in Section 3.2 compares the extracted answer to these labels, so a systematic mislabel trains the model to match wrong targets; and the outcome criteria in Figure 6 require contradictory variants to have inverse truth values, so the headline 47.22 vs 42.90 result and all Table 4/5 outcome numbers are scores against LLM-generated labels rather than verified mathematical truth. Section 4.3's case study is consistent with this: the GPT-4o process judge endorses a proof whose key claim about preperiodic points is unsupported, showing the LLM-only validation loop does not guarantee soundness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DeepTheorem, a dataset of 121K informal mathematical theorems with LLM-generated proofs and annotations, together with theorem variants for binary-reward RL. It proposes RL-Zero training using GRPO with rewards based on whether the model's proved/disproved classification matches the variant labels, and an evaluation framework with outcome and process metrics. Experiments on FIMO, HMMT, and Putnam compare DeepTheorem against OpenR1-Math-Proof and several commercial/open-source LLMs, reporting that RL-Zero with DeepTheorem improves over SFT and achieves state-of-the-art results.","tokens_in":25468,"tokens_out":9886,"duration_ms":86323,"significance":"The paper addresses an important problem—scaling informal theorem proving without formal proof assistants—and the release of code and data is a useful community resource. However, the headline claims currently rest on unverified LLM-generated labels, a metric that coincides with the RL reward, and a process judge that demonstrably accepts a proof with a false claim. With independent validation (human audit of variants, a human-verified test subset, multiple seeds, and corrected claims about state-of-the-art), the contributions could be significant; as presented, the evidence is not yet convincing.","major_comments":[{"comment":"The ground-truth labels for the variant construction in Section 3.1 are not verified. The prompt in Appendix A.8 instructs the model to 'negate the original statement' with minimal changes, but for statements of the form ∀x(H(x)→C(x)), negating only the conclusion yields ∀x(H(x)→¬C(x)), which is not the logical negation and can be vacuously true. Table 2's pair x>1 and x<1 is not contradictory, since for x=1 both statements are false. These labels are used both as the RL reward (Section 3.2) and as the outcome criteria (Figure 6, conditions 3–4), so a systematic mislabeling would train models to match wrong targets and make the reported numbers in Tables 4 and 5 measure agreement with an unverified distribution rather than mathematical truth. The paper reports no human audit, no error-rate analysis, and no inter-annotator agreement for the variant labels or the 'completeness annotation' described in Section 3.1.","section":"Section 3.1, Figure 6"},{"comment":"The outcome metric in Figure 6 is the same signal that RL-Zero optimizes: the reward function in Section 3.2 compares the extracted \\boxed{proved}/\\boxed{disproved} answer with the variant labels, and the outcome criteria require exactly the same consistency pattern across original, entailing, and contradictory variants. Consequently, the RL-vs-SFT comparisons in Table 4 partly measure how well the model has learned the training objective, not an independent property of theorem-proving ability. To support the claim of improved reasoning, the authors should evaluate on a human-verified test set or on variants whose truth values are formally checked, and report the correlation between the variant-consistency metric and a genuinely independent correctness signal.","section":"Section 3.2 vs. Section 3.3"},{"comment":"The case study is direct evidence that the LLM-based process evaluation is unsound. The model's solution asserts that 'the number of integer preperiodic points of P is at most n for each iterate,' which is false; for example, P(x)=x^2-1 has the three integer preperiodic points {-1,0,1} although n=2. The paper's own analysis praises this reasoning as 'clear and correct,' and the GPT-4o judge in Appendix A.10 would assign high sub-scores because it does not check the mathematical validity of this step. This shows that the process scores reported in Tables 4 and 5 cannot be interpreted as proof quality.","section":"Section 4.3"},{"comment":"The data construction pipeline is entirely LLM-driven: theorem justification, rationale summarization, proof generation by o3-mini, logical validation, difficulty annotation, and topic annotation each use LLM outputs with no human audit or reported error rates. The claims that the 121K theorems are 'high-quality' and 'IMO-level' rest on these unverified annotations, yet no sample is shown to have been human-checked. Similarly, the decontamination procedure in Section 2.1 uses an embedding similarity threshold of 0.7 and an LLM 'contamination justification' step, but no precision/recall evaluation is given for this filter, making the 'strict decontamination' claim unquantified.","section":"Section 2.1"},{"comment":"The experimental comparison is based on single runs with no random seeds, no confidence intervals, and no significance tests. Several reported differences are small (e.g., 1.5B RL outcome: 35.85 vs 35.69; 3B RL outcome: 38.53 vs 38.65), so the assertion in Section 4.2 that 'our RL-Zero training paradigm consistently outperforms SFT' is not statistically supported. The paper should report multiple seeds and variability or refrain from claiming consistent superiority.","section":"Table 4"},{"comment":"The claim that 'our 7B model ... outperforms SOTA models of much larger sizes' is contradicted by the paper's own Table 5. The DeepTheorem-RL-7B average outcome score is 47.22, whereas o1, Gemini2.5-Pro, and o3-mini score 58.88, 59.78, and 68.28 respectively. The abstract's 'state-of-the-art accuracy' is therefore not supported; the accurate statement is that the 7B model outperforms several larger open-weight models, not the best commercial systems.","section":"Section 4.2, Table 5"}],"minor_comments":[{"comment":"The text says 'we manually expand each question in the three data sources into multiple entailing or contradictory variants,' but Section 3.1 and Appendix A.8 describe an LLM-driven expansion using Qwen2.5-72B-Instruct; please clarify who performs the expansion and whether any human checks the resulting variants.","section":"Section 3.3"},{"comment":"The difficulty scale is inconsistent: Section 2 mentions 'level 5~10', Section 2.1 says 'a scale of 1 to 9', and Appendix A.6 instructs annotators to rate 'between 1 and 10, with increments of 0.5'. Please reconcile these statements.","section":"Section 2.1 vs. Appendix A.6"},{"comment":"The process evaluation weights (0.4 validity, 0.3 completeness, 0.2 correctness, 0.1 clarity) are introduced without justification or sensitivity analysis; since process scores are reported as primary results in Table 4, the weights should be justified or shown not to affect conclusions.","section":"Appendix A.10"},{"comment":"There is a grammatical error in the paragraph beginning 'DeepTheorem achieves strong parameter efficiency': 'We demonstrates' should be 'We demonstrate'.","section":"Section 4.2"},{"comment":"The citation for the Harvard-MIT Mathematics Tournament uses an inconsistent format with two years and a bare URL; please format it according to the journal's reference style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a dataset/benchmark paper with a training method. The overclaim about state-of-the-art performance and the complete lack of human verification of the ground-truth labels are serious. I would encourage the editor to request a revised version with a human-validated evaluation subset, multiple seeds, and corrected claims before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — quick read on arXiv:2505.23754 (DeepTheorem). The paper is worth your attention for two reasons: it ships a large (121K) informal theorem-proving dataset with annotations, and it introduces a clean RL idea — use LLM-generated entailing and contradictory variants of theorems to give binary rewards for informal proofs, without a formal proof assistant. That's a genuine new training paradigm, and the resource is real and public. I'd use the dataset as a training/eval resource.\n\nBut the paper's central claims outrun its evidence. The headline 'state-of-the-art accuracy' is not supported by Table 5: their best 7B model scores 47.22 average, behind o3-mini (68.28), Gemini 2.5 Pro (59.78), o1 (58.88), o1-mini (52.46). At best they show parameter efficiency relative to much larger models, and even that is limited to the reported runs.\n\nThe deeper problem is that the outcome metric (Fig. 6) is the same signal used as the RL reward, and both depend on LLM-generated labels with no human audit. The stress-test note is right that 'negating' a theorem like x>1 to x<1 is not always a true contradiction; if the hypotheses don't exclude x=1, the variant isn't false. If the variant labels are systematically wrong, both the reward and the evaluation are measuring agreement with a self-generated distribution, not mathematical truth. The paper's own case study (Section 4.3) is direct evidence: the proof the authors and GPT-4o endorse as 'correct' contains a bogus claim about preperiodic points bounding fixed points by n per iterate. So the process metric is demonstrably lenient.\n\nWhat holds up: Table 4 shows RL beats SFT in almost all comparisons, and DeepTheorem beats the OpenR1-Math-Proof baseline on average, at least in these single runs. That's a useful within-subfield result, but it's not the SOTA story, and we have no seeds, no significance tests, no human audit of the labels.\n\nBottom line: this deserves a serious referee, not a desk reject — the resource and method are valuable and the problems are fixable (human audit on a sample, multiple seeds, reworded claims). But as it stands, the abstract overstates, and the evaluation loop needs an independent check. I'd send it to review with major-revision expectations.","headline":"A genuinely useful dataset and RL reward idea, but the headline SOTA claims and the LLM-only label loop need serious revision before the paper is publishable as-is.","tokens_in":26139,"tokens_out":2973,"would_cite":true,"duration_ms":28112,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that informal theorem proving can be advanced at scale by mining 121K IMO-level theorem–proof pairs, checking them with LLMs, and training with RL-Zero on guaranteed true/false variants, so a 7B model outscores larger…","keywords":["informal theorem proving","reinforcement learning","RL-Zero","LLM reasoning","theorem variants","mathematical benchmark","GRPO","natural language proofs"],"falsifier":"A human audit of a random sample of DeepTheorem theorem–proof–variant triples: expert mathematicians would verify each theorem's truth value, each proof's validity, and each variant's entailment or contradiction relation. If a nontrivial fraction of labels are wrong, the RL reward is noisy and the outcome scores are partly scored against a faulty key; a second check is to generate new variants with a different LLM after training and see whether the model's consistency survives the distribution shift.","tokens_in":24995,"feed_emoji":"🧮","tokens_out":12757,"duration_ms":103384,"temperature":0.7,"pith_summary":"DeepTheorem claims that large language models can get better at proving hard mathematical theorems without a formal proof assistant, by training on natural-language theorem–proof pairs and by rewarding consistent answers on deliberately modified theorem variants. The paper builds a 121K-sample dataset of IMO-level statements, each with a correctness label, a difficulty score, a topic tag, and a proof written by an advanced LLM, plus entailing and contradictory variants of each theorem whose truth value is guaranteed by the original. On that data it runs RL-Zero with a binary reward: the model must end by saying proved or disproved, and the variant's known truth value decides the reward. Across three olympiad-level benchmarks, RL-Zero on DeepTheorem beats supervised fine-tuning and the prior theorem-proving dataset baseline, with a 7B model reaching 47.22 average outcome score and 34.04 process score. The paper takes this as evidence that scalable, web-mined informal theorem proving is a viable path to stronger mathematical reasoning in LLMs.","feed_headline":"121K web-mined theorems plus RL beat SFT on proof tests","feed_subtitle":"A 7B model scores 47.22 average versus 42.90 baseline and outranks larger rivals.","key_machinery":"The load-bearing object is the set of entailing and contradictory theorem variants. For each original theorem, a variant is produced by a minimal change—for example, from $x>1$ to $x>0$ (entailing) or to $x<1$ (contradictory)—so that one variant is guaranteed true and the other guaranteed false whenever the original theorem has a definite truth value. This gives the RL training a binary outcome signal: GRPO, a policy-gradient algorithm, rewards the model with 1 when its proved/disproved answer matches the variant's known truth value and 0 otherwise, plus sanity checks against degenerate outputs. The same variant structure defines the outcome evaluation, which counts a test case as passed only when the original theorem and all entailing variants get the same truth value and all contradictory variants get the inverse.","core_discovery":"The central discovery, as the authors state it, is that a theorem statement for training and evaluation does not have to be true: a false statement can be disproved, and both directions can be scored. They therefore transform each theorem into variants by minimal logical edits, some entailed by the original and some contradictory to it, so the correct answer is known from the original theorem alone without checking a proof. This converts informal theorem proving into a binary-reward reinforcement-learning problem, and their outcome metric requires the model to assign consistent truth values across all variants. The paper reports that this scheme, trained from a base 7B model with GRPO, achieves 55.56 outcome on FIMO, 28.81 on HMMT, and 57.29 on PutnamBench, an average of 47.22, above the prior theorem-proving dataset baseline (42.90 outcome) and above every open-source model they compared, while ranking behind only three larger commercial models.","pith_inferences":["A testable extension the paper leaves implicit is whether the variant-consistency signal alone, without the original proof text, accounts for the gain; ablating proofs from training would separate the effect of the RL reward from the effect of seeing high-quality proofs.","The outcome metric could be gamed by a model that always outputs the truth value of the original theorem for every variant; scoring on fresh variants generated after training would make the metric more robust and is a natural next experiment.","The paper does not report an error-rate analysis of the LLM-generated labels, variants, or judge scores, and its own case study praises a proof with a shaky intermediate claim; a human audit of a random sample would clarify whether the reported outcome score reflects mathematical validity or agreement with the LLM-generated answer key."],"forward_implications":["If the central claim is correct, a 7B model trained with RL-Zero on DeepTheorem outperforms supervised fine-tuning on the same corpus at every model size tested, so RL-Zero rather than more SFT data is the lever for informal theorem-proving skill.","The improvement transfers across FIMO, HMMT, and PutnamBench, so the result is not an artifact of one benchmark's style.","Because the reward is binary and derivable from logical relations between statements, the same variant-consistency recipe could be applied to other domains where statements can be generated in entailing and contradictory pairs.","Process scores also improve alongside outcome scores, suggesting the trained model produces more complete, logically coherent proof traces, not just better final answers."],"supporting_citations":[{"why":"OpenR1-Math is the existing theorem-proving corpus that, after the paper's processing pipeline, becomes the dataset baseline for both SFT and RL training.","marker":"Face, 2025"},{"why":"Supplies GRPO, the reinforcement-learning algorithm used for RL-Zero training.","marker":"Shao et al., 2024"},{"why":"Establishes the RL-Zero paradigm and the reasoning-trace style that the paper's training recipe builds on.","marker":"DeepSeek-AI et al., 2025"},{"why":"Provides the RL training configuration, including no KL regularization and no entropy loss.","marker":"Hu et al., 2025"},{"why":"FIMO is one of the three evaluation benchmarks for outcome and process scores.","marker":"Liu et al., 2023"},{"why":"PutnamBench is another evaluation benchmark for outcome and process scores.","marker":"Tsoukalas et al., 2024"},{"why":"HMMT is the third benchmark, from which a theorem-proving subset is constructed for evaluation.","marker":"Harvard-MIT Mathematics Tournament, 2024, 2025"},{"why":"The Qwen2.5 base models are the backbones used for both SFT and RL-Zero training.","marker":"Yang et al., 2024"},{"why":"The difficulty-annotation prompting strategy used to assign each dataset theorem a score on the 1–9 scale.","marker":"Gao et al., 2025"}],"fun_headline_variants":["False theorem variants turn proof into RL reward, beating SFT","RL from false theorems lifts 7B model past larger open rivals","Consistent truth scoring on theorem variants beats SFT","DeepTheorem: false variants as reward signals, 7B beats SFT baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated labels are mathematically correct at every layer: the LLM's True/False validation of each theorem–proof pair, the advanced LLM's proofs, and the 72B model's entailing and contradictory variants all preserve real mathematical truth, so the binary reward and the evaluation scores measure validity rather than agreement with an LLM-generated answer key.","fun_headline_variants_meta":{"raw":{"variants":["False theorem variants turn proof into RL reward, beating SFT","RL from false theorems lifts 7B model past larger open rivals","Consistent truth scoring on theorem variants beats SFT","DeepTheorem: false variants as reward signals, 7B beats SFT baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3075,"prompt_tokens":951,"completion_tokens":2124,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2050}},"tokens_in":567,"tokens_out":2124,"duration_ms":14188,"temperature":1.0,"reasoning_tokens":2050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:39:26.565547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human audit of a random sample of DeepTheorem theorem–proof–variant triples: expert mathematicians would verify each theorem's truth value, each proof's validity, and each variant's entailment or contradiction relation. If a nontrivial fraction of labels are wrong, the RL reward is noisy and the outcome scores are partly scored against a faulty key; a second check is to generate new variants with a different LLM after training and see whether the model's consistency survives the distribution shift.","supporting_citations":[],"review_version":1}