REVIEW 2 cited by
From Informal to Formal -- Incorporating and Evaluating LLMs on Natural Language Requirements to Verifiable Formal Proofs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The research in AI-based formal mathematical reasoning has shown an unstoppable growth trend. These studies have excelled in mathematical competitions like IMO and have made significant progress. This paper focuses on formal verification, an immediate application scenario of formal reasoning, and breaks it down into sub-tasks. We constructed 18k high-quality instruction-response pairs across five formal specification languages (Coq, Lean4, Dafny, ACSL, and TLA+) by distilling gpt-4o and evaluated against ten open-sourced LLMs, including recent popular DeepSeek-R1. We also fine-tuned several 7~8B small models to achieve comparable performance with Deepseek-R1-671B. Interestingly, we observed that fine-tuning with formal data also enhances mathematics, reasoning, and coding capabilities. Fine-tuned models are released at https: //huggingface.co/fm-universe.
Forward citations
Cited by 2 Pith papers
-
Requirements Development and Formalization for Reliable Code Generation: A Multi-Agent Vision
The paper proposes ReDeFo, a multi-agent pipeline that uses formal specifications and verification to generate reliable code from natural language requirements.
-
CodeGrad: Integrating Multi-Step Verification with Gradient-Based LLM Refinement
CodeGrad improves code generation by routing structured critic feedback into up to two refinement rounds, but its formal-verification claim rests on AI-written proofs judged by the AI.
Discussion (0). Sign in to comment.