Computation and Language · arXiv:2611.16433 v1 · submitted 12 Nov 2026 · 28 references
daVinci-Prover: Agent-Native Mid-Training for Formal Theorem Proving
The paper says
“Advances in large language models (LLMs) are rapidly transforming formal theorem proving. Yet further progress remains constrained by the lack of a transparent and effective data recipe for systematically improving theorem-proving models.”
more of the abstract
In this work, we argue that effective agentic provers require strong mathematical skill priors to ground theorem proving and agentic trajectories to enable long-horizon, interactive proof construction. To this end, we synthesize an agent-native 33.7B-token mid-training corpus covering decomposition, search, self-verification, and revision, together with verifier-grounded agentic trajectories. We further introduce the daVinci-Prover harness, which enables models to construct proofs and interact with a Lean verifier in an isolated workspace, and use it to collect approximately 36K high-quality agent trajectories for supervised fine-tuning. Our mid-training recipe yields an average improvement of 13.7 points over the matched Qwen3-8B SFT-only baseline across MiniF2F, FATE-M, and ProofNet. When applied to the Qwen3.5 series, our approach also yields strong performance across theorem-proving benchmarks and achieves near-saturating MiniF2F performance with memory compaction. The results highlight agent-native mid-training as an effective foundation for developing stronger interactive theorem-proving agents.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
A competent, well-executed empirical paper on mid-training for theorem proving, but the novelty is incremental and the decontamination concern is worth watching. This line is wrong
the referee pass did not run
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 28 references say.
What would settle it
Apply a fuzzy deduplication step on the mid-training corpus against all evaluation benchmarks: use 13-gram overlap (after normalization) on the problem statement, and flag any parent sample with >80% Jaccard similarity. Remove all such flagged samples and rerun the Qwen3-8B-Base + Midtrain + SFT evaluation for MiniF2F, FAE-M, and ProoNet. If the average pass@1 does not drop by more than 1 point, the exact-match procedure is likely sufficient; if it drops substantially (e.g., >5 points), then decontamination is incomplete and the reported gains are inflated.
from the skeptic's reading
How the review went
3 readings by deepseek-v4-flash · 13 Nov 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
Not run on this paper: the referee, the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a solid engineering contribution. The paper demonstrates that a carefully constructed mid-training corpus combining mathematical skill priors (decomposition, search, revision, self-verification) with environment-native agent trajectories consistently improves formal theorem-proving performance across multiple benchmarks and model scales. The 13.7-point pass@1 gain over the Qwen3-8B SFT baseline is real and well-ablated. The authors also provide a clean harness, detailed data construction pipelines, and thoughtful analyses of agent behavior. Credit where due: this is the most transparent and systematic mid-training recipe for theorem proving I have seen, even if the individual components are not new.
The soft spots are real but not fatal. The novelty is incremental — mid-training for reasoning tasks has been explored elsewhere, and the data synthesis techniques (decomposition from DeepSeek-Prover, search from OProver, etc.) are adapted from prior work. The paper's main claim is empirical, and the evidence is strong enough that I wouldn't call it a flaw, but readers should calibrate their expectations.
More concretely, the decontamination is the one thing that keeps me from being fully convinced. The corpus contains 33.7B tokens scraped from sources like AoPS and Lean Workbook that overlap heavily with olympiad-level problems. The decontamination uses exact matching only, and the authors reserve fuzzy matches for manual review but do not report those results. The 0.034% overlap figure only counts verbatim copies. If even a few dozen reformulated problems slip through, the gain on MiniF2f or ProofNet could be inflated. That said, the consistent improvement across multiple benchmarks (including Putnam, which is less likely to overlap) partially mitigates this concern. A careful reader should decide their own threshold of trust.
Who is this for? People building neural theorem provers or working on training data recipes for reasoning. It will be cited by follow-ups in this thin slice of the field. The authors are honest about limitations (not replaying generlization data, reliance on a strong teacher model) and they plan to release artifacts, which is commendable.
I would send this out for peer review. It is not a landmark paper, but it is a competently executed study with enough rigor to let referees evaluate its claims. The decontamination issue is a standard question that the review process can handle.
The strongest objection
The paper’s central claim of a 13.7-point pass@1 improvement depends on the assumption that the evaluation benchmarks (MiniF2F, FATE-M, ProofNet) are not leaked into the mid-training corpus. The decontamination procedure (Appendix C) uses exact matching after Unicode normalization, tokenization, and anonymization, requiring at least 20 tokens and 80 characters. While this catches verbatim copies, it does not address near-duplicates or semantically equivalent problem statements that have been lightly reformulated or the statement permuted. The reported 0.034% overlap only counts exact matches; if even a few dozen problems are leaked via fuzzy similarity, the observed gain could be partly or entirely spurious. The authors acknowledge the limitation by reserving fuzzy matches for manual review, but no such review is reported. Given that the mid-training data sources (AoPS, Lean Workbook, etc.) contain many olympiad-level problem statements similar to those in MiniF2F and ProofNet, this is a plausible threat to the core result.
The reader’s weakest_assumption identifies exctly the same concen: deconamination may be insufficient due to exct-match only. This is indeed the most load-bearing concern becus the eire 13.7-point image rests on the belief that no eval problems have seped into training. The paper’s own limation statmen (Appndix C) acknowledges that fuzzy matches go to manual review but does not report doing tht review. My proposd concte tst would settl whther th can is realized. I do not mnd the vdict bcuse th pper’s multiple benchmarks and consistent improvements across scalers all likly surviv even a modest leake, unless th leak is s systematic. The reader’s ACCEPT s jud with MODRAE conidenc s rasonabl, d the suggetd tst woud either confirm or repir th main claim witout changing the vdict now.
Pith's reading of the paper (a machine's summary, not the authors')
The authors argue that effective automated theorem provers need both strong mathematical reasoning skills and experience interacting with a proof assistant (Lean) in real time. To teach both, they build a large mid-training corpus containing four types of skill data: decomposing problems into lemmas, searching for proof steps, diagnosing and fixing errors, and self-verifying reasoning. They also collect agentic trajectories by running a strong teacher model in the daVinci-Prover harness, which lets the model edit files, retrieve mathlib declarations, and verify Lean code in a sandboxed workspace. These trajectories show the model how to use tools and respond to error messages. After this mid-training, they fine-tune the model on about 36,000 successful proof sessions from the teacher. Evaluated on five Lean 4 benchmarks, the approach consistently outperforms training without mid-training, and it often improves by larger margins on harder problems. The results suggest that mid-training tailored to the interactive proving task provides a stronger starting point for later supervised fine-tuning.
Core claim. Agent-native mid-training yields an average improvement of 13.7 pass@1 points over the matched Qwen3-8B SFT-only baseline across MiniF2F, FATE-M, and ProofNet (abstract, Section 5).
The premise it rests on. The decontamination procedure (exact matching after Unicode normalization, tokenization, and anonymization, requiring at least 20 tokens and 80 characters) is sufficient to prevent data leakage from the evaluation benchmarks into the 33.7B-token mid-training corpus. The paper reports only 0.034% token overlap, but this relies on exact matching; fuzzy similarity or reformulated problems could still leak and inflate benchmark performance (Appendix C).
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Advances in large language models (LLMs) are rapidly transforming formal theorem proving. Yet further progress remains constrained by the lack of a transparent and effective data recipe for systematically improving theorem-proving models. In this work, we argue that effective agentic provers require strong mathematical skill priors to ground theorem proving and agentic trajectories to enable long-horizon, interactive proof construction. To this end, we synthesize an agent-native 33.7B-token mid-training corpus covering decomposition, search, self-verification, and revision, together with verifier-grounded agentic trajectories. We further introduce the daVinci-Prover harness, which enables models to construct proofs and interact with a Lean verifier in an isolated workspace, and use it to collect approximately 36K high-quality agent trajectories for supervised fine-tuning. Our mid-training recipe yields an average improvement of 13.7 points over the matched Qwen3-8B SFT-only baseline across MiniF2F, FATE-M, and ProofNet. When applied to the Qwen3.5 series, our approach also yields strong performance across theorem-proving benchmarks and achieves near-saturating MiniF2F performance with memory compaction. The results highlight agent-native mid-training as an effective foundation for developing stronger interactive theorem-proving agents.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-11-13
The authors' main claim, as the classifier picked it out 1.00
“The results highlight agent-native mid-training as an effective foundation for developing stronger interactive theorem-proving agents.”
How the main claim is stated: asserted 0.96
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.18 · human participants 0.05 · controlled comparison 0.96 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.08
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Advances in large language models (LLMs) are rapidly transforming formal theorem proving. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| Yet further progress remains constrained by the lack of a transparent and effective data recipe for systematically improving theorem-proving models. | gap 0.99 | 0.08 | 0.03 | asserted 1.00 |
| In this work, we argue that effective agentic provers require strong mathematical skill priors to ground theorem proving and agentic trajectories to enable long-horizon, interactive proof construction. | background 0.45 | 0.16 | 0.15 | asserted 0.93 |
| To this end, we synthesize an agent-native 33.7B-token mid-training corpus covering decomposition, search, self-verification, and revision, together with verifier-grounded agentic trajectories. | method 0.82 | 0.97 | 0.07 | asserted 1.00 |
| We further introduce the daVinci-Prover harness, which enables models to construct proofs and interact with a Lean verifier in an isolated workspace, and use it to collect approximately 36K high-quality agent trajectories for supervised fine-tuning. | method 0.88 | 0.97 | 0.04 | asserted 1.00 |
| Our mid-training recipe yields an average improvement of 13.7 points over the matched Qwen3-8B SFT-only baseline across MiniF2F, FATE-M, and ProofNet. | result 1.00 | 0.96 | 0.14 | asserted 1.00 |
| When applied to the Qwen3.5 series, our approach also yields strong performance across theorem-proving benchmarks and achieves near-saturating MiniF2F performance with memory compaction. | result 1.00 | 0.95 | 0.10 | asserted 1.00 |
| The results highlight agent-native mid-training as an effective foundation for developing stronger interactive theorem-proving agents. | interpretation 1.00 | 0.77 | 0.89 | asserted 0.96 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.98
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 6,356 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (11)
Figures are the authors', reproduced from the paper.
References (28)
- [4] Jui-Hui Chung, Ziyang Cai, Zihao Li, Qishuo Yin, Rohit Agarwal, Simon Park, Rodrigo Porto, Narutatsu Ri, Ziran Yang, Shange Tang, et al. Goedel-architect: Streamlining formal theorem proving with blueprint generation and refinement.arXiv preprint arXiv:2606.06468, on PithPith's reviewarXiv
- [6] Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia Li, Mengzhou Xia, Danqi Chen, Sanjeev Arora, et al. Goedel-prover: A frontier model for open-source automated theorem proving.arXiv preprint arXiv:2502.07640, on PithPith's reviewarXiv
- [7] Yong Lin, Shange Tang, Bohan Lyu, Ziran Yang, Jui-Hui Chung, Haoyu Zhao, Lai Jiang, Yihan Geng, Jiawei Ge, Jingruo Sun, et al. Goedel-prover-v2: Scaling formal theorem proving with scaffolded data synthesis and self-correction. InInternational Conference on Learning Represen- ... on Pith
- [8] 10 daVinci-Prover: Agent-Native Mid-Training for Formal Theorem Proving Junqi Liu, Zihao Zhou, Zekai Zhu, Marco Dos Santos, Weikun He, Jiawei Liu, Ran Wang, Yun- zhou Xie, Junqiao Zhao, Qiufeng Wang, et al. Numina-lean-agent: An open and general agentic reasoning system for fo... on PitharXiv
- [9] David Ma, Kaijing Ma, Shawn Guo, Yunfeng Shi, Enduo Zhao, Jiajun Shi, Zhaoxiang Zhang, Gavin Cheung, Jiaheng Liu, and Zili Wang. Oprover: A unified framework for agentic formal theorem proving.arXiv preprint arXiv:2605.17283, on PithPith's reviewarXiv
- [12] ZZ Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, et al. Deepseek-prover-v2: Advancing formal mathematical rea- soning via reinforcement learning for subgoal decomposition.arXiv preprint arXiv:2504.21801, on PithPith's reviewarXiv
- [13] Borja Requena, Austin Letson, Krystian Nowakowski, Izan Beltran-Ferreiro, and Leopoldo Sarra. A minimal agent for automated theorem proving.arXiv preprint arXiv:2602.24273, on PithPith's reviewarXiv
- [14] Zhihong Shao, Yuxiang Luo, Chengda Lu, ZZ Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Deepseekmath-v2: Towards self-verifiable mathematical reasoning.arXiv preprint arXiv:2511.22570, on PitharXiv
- [15] Ziju Shen, Naohao Huang, Fanyi Yang, Yutong Wang, Guoxiong Gao, Tianyi Xu, Jiedong Jiang, Wanyi He, Pu Yang, Mengzhou Sun, et al. Real-prover: Retrieval augmented lean prover for mathematical reasoning.arXiv preprint arXiv:2505.20613, on PitharXiv
- [16] Chengying Tu, Xuemiao Zhang, Rongxiang Weng, Rumei Li, Chen Zhang, Yang Bai, Hongfei Yan, Jingang Wang, and Xunliang Cai. A survey on llm mid-training.arXiv preprint arXiv:2510.23081, on PitharXiv
- [17] Sumanth Varambally, Thomas V oice, Yanchao Sun, Zhifeng Chen, Rose Yu, and Ke Ye. Hilbert: Recursively building formal proofs with informal reasoning. InInternational Conference on Learning Representations, volume 2026, pp. 38998–39031, on Pith
- [18] Haiming Wang, Mert Unsal, Xiaohan Lin, Mantas Baksys, Junqi Liu, Marco Dos Santos, Flood Sung, Marina Vinyes, Zhenzhe Ying, Zekai Zhu, et al. Kimina-prover preview: Towards large formal reasoning models with reinforcement learning.arXiv preprint arXiv:2504.11354, 2025a. Jianin... on PithPith's reviewarXiv
- [19] Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Mid-training incentivizes reinforcement learning scaling.arXiv preprint arXiv:2506.20512, 2025b. Ran Xin, Chenguang Xi, Jie Yang, Feng Chen, Hang Wu, Xia Xiao, Yifan Sun, Shen Zheng, and Ming Ding. Bfs-prover: S... on PithPith's reviewarXiv
- [20] 11 daVinci-Prover: Agent-Native Mid-Training for Formal Theorem Proving Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence.... on PitharXiv
- [21] Zonghan Yang, Shengjie Wang, Kelin Fu, Wenyang He, Weimin Xiong, Yibo Liu, Yibo Miao, Bofei Gao, Yejie Wang, Yanhao Li, et al. Kimi-dev: Agentless training as skill prior for swe-agents. In International Conference on Learning Representations, volume 2026, pp. 148369–148436, on Pith
- [22] Ji Zeng, Dayuan Fu, Tiantian Mi, Yumin Zhuang, Yaxing Huang, Xuefeng Li, Lyumanshan Ye, Muhang Xie, Qishuo Hua, Zhen Huang, et al. davinci-dev: Agent-native mid-training for software engineering.arXiv preprint arXiv:2601.18418, on PitharXiv
- [23] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and rl on reasoning language models.arXiv preprint arXiv:2512.07783, on PitharXiv
- [24] Xueliang Zhao, Wenda Li, and Lingpeng Kong. Decomposing the enigma: Subgoal-based demon- stration learning for formal theorem proving.arXiv preprint arXiv:2305.16366, on PithPith's reviewarXiv
- [25] Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: a cross-system benchmark for formal olympiad-level mathematics.arXiv preprint arXiv:2109.00110, on PithPith's reviewarXiv
- [26] Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu. Clean–eval: Clean evaluation on contaminated large language models. InFindings of the Association for Computational Linguistics: NAACL 2024, pp. 835–847, on Pith
show 8 more
- [27] LIMITATION During the mid-training phase, we do not replay any generalization data. However, we believe that incorporating generalization replay is both necessary and important for maintaining and improving the model’s generalization capabilities. A RELATEDWORK Mid-training.Re... on Pith
- [28] For Lean data, we extract the theorem signature by removing nested comments and proof bodies, normalize Unicode characters, tokenize the remaining statement, and anonymize the declaration name. We then perform exact matching against all available benchmark variants. To avoid f... on Pith
- [2015] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cogni- tive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, on PithPith's reviewarXiv
- [2021] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 31932–31951, on Pith
- [2023] Kaito Baba, Chaoran Liu, Shuhei Kurita, and Akiyoshi Sannai. Prover agent: An agent-based framework for formal mathematical proofs.arXiv preprint arXiv:2506.19923, on PitharXiv
- [2024] Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie F Ruan, Tyler Griggs, Alexander Glenn Shaw, Hritik Bansal, et al. Openthoughts-agent: Data recipes for agentic models.arXiv preprint arXiv:2606.24855, on PithPith's reviewarXiv
- [2025] Jiangjie Chen, Wenxiang Chen, Jiacheng Du, Jinyi Hu, Zhicheng Jiang, Allan Jie, Xiaoran Jin, Xing Jin, Chenggang Li, Wenlei Shi, et al. Seed-prover 1.5: Mastering undergraduate-level theorem proving via learning from experience.arXiv preprint arXiv:2512.17260, on PitharXiv
- [2026] Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathemat- ics.arXiv preprint arXiv:2302.12433, on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment