{"id":"6f35b892-c887-4b74-98d1-499e026ca3dc","arxiv_id":"2606.19399","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"VERITAS improves zero-shot formal theorem proving to 40.6% on miniF2F and 7.3% on a new combinatorics benchmark by routing verifier signals into a two-phase Best-of-N then critic-guided MCTS protocol.","lead":"VERITAS uses a two-phase protocol of Best-of-N sampling followed by critic-guided MCTS that feeds detailed verifier signals like syntax errors and partial progress back into search as negative examples. A smart generalist might read it to see how richer tool feedback can improve LLM reasoning on formal math without any task-specific training.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Gains on miniF2F and CombiBench hinge on critic reliably converting Phase-1 failure signals into unbiased negative examples for MCTS without erasing net improvement.","rationale":"Reader’s weakest assumption matches the central dependency exactly. Because the manuscript was reviewed from abstract only, the same assumption remains the least-secured link once the full text is examined; no stronger internal inconsistency appears from the provided claim structure.","tokens_in":1742,"tokens_out":392,"duration_ms":12576,"concrete_test":"Re-run the miniF2F and CombiBench evaluations with an ablated Phase 2 that performs the same number of MCTS iterations but replaces the critic with uniform random negative-example sampling; if the solve count drops to or below the Phase-1 Best-of-N baseline, the critic’s signal-to-guidance mapping is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The protocol claims Phase 2 adds solves (40.6% vs 36.9% Best-of-5 on miniF2F; 7.3% vs 1.8%/3.6% on 55-theorem CombiBench) precisely because the critic ingests verifier signals (syntax errors, type mismatches, partial goals) as explicit negatives to steer MCTS. This requires that (a) the critic mapping from raw verifier output to search guidance is stable and non-biased across proof states, and (b) any extra exploration cost does not offset the reported delta. If the critic over-penalizes viable branches or the negative-example construction introduces systematic preference for certain lemma styles, the headline attribution to “feedback-driven exploration” fails even while Phase-1 solves are preserved. The abstract-only review already flagged this; full text would need to show the exact critic prompt, training regime (if any), and an ablation removing the critic while keeping search budget matched.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents VERITAS, a zero-shot LLM-based formal theorem proving framework using a two-phase protocol: Best-of-N sampling followed by critic-guided MCTS that ingests Phase-1 verifier failure signals (syntax errors, type mismatches, partial goals) as explicit negative examples. The protocol is designed to preserve all Phase-1 solves so that any additional solves in Phase 2 are attributable to feedback-driven exploration. It reports 40.6% on miniF2F (vs. independently run Best-of-5 at 36.9% and Portfolio at 26.2%) and 7.3% on the released 55-theorem VERITAS-CombiBench (vs. Best-of-5 at 1.8% and Portfolio at 3.6%), with GitHub artifacts provided.","tokens_in":1945,"tokens_out":535,"duration_ms":15662,"significance":"If the central performance claims hold, the work demonstrates the value of routing rich verifier signals into search rather than collapsing them to binary outcomes, particularly in combinatorics where iterative lemma recovery benefits from guided exploration. The release of VERITAS-CombiBench and the preservation of Phase-1 solves are concrete strengths that enable direct attribution and community reuse.","major_comments":[{"comment":"The description of the critic (how verifier outputs are mapped to negative examples for MCTS) lacks an ablation that removes the critic while exactly matching Phase-2 search budget; without this, the attribution of the 3.7-point miniF2F gain and the CombiBench delta specifically to feedback-driven exploration cannot be isolated from additional search effort.","section":"Methods (Phase-2 protocol)"},{"comment":"No details are provided on the critic's prompt template, whether it is zero-shot or fine-tuned, or how negative-example construction avoids systematic bias across proof states (e.g., over-penalizing viable branches with certain lemma styles); this directly affects the weakest assumption that the critic reliably improves exploration without erasing net gains.","section":"Experimental setup and critic implementation"}],"minor_comments":[{"comment":"The abstract states that artifacts are available on GitHub; the manuscript should include a direct link and a brief description of what is released (models, prompts, benchmark theorems).","section":"Abstract"},{"comment":"Table or figure reporting per-theorem results on CombiBench would help readers assess whether the 7.3% aggregate reflects broad improvement or concentration on a few theorems.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback and for highlighting the strengths of our work, including the preservation of Phase-1 solves and the release of VERITAS-CombiBench. We respond point-by-point to the major comments below and commit to revisions that directly address the concerns raised.","responses":[{"response":"We agree that an ablation isolating the critic's contribution while exactly matching the Phase-2 search budget would strengthen attribution of the observed gains. In the revised manuscript we will add this ablation: Phase-2 will be re-run with the identical number of MCTS simulations and node expansions but without the critic (using uniform selection or a non-feedback baseline), and we will report the resulting solve rates on both miniF2F and VERITAS-CombiBench. This will allow direct comparison and clarify how much of the 3.7-point and CombiBench deltas are due to feedback-driven exploration versus additional search effort.","revision_made":"yes","referee_comment":"[Methods (Phase-2 protocol)] The description of the critic (how verifier outputs are mapped to negative examples for MCTS) lacks an ablation that removes the critic while exactly matching Phase-2 search budget; without this, the attribution of the 3.7-point miniF2F gain and the CombiBench delta specifically to feedback-driven exploration cannot be isolated from additional search effort."},{"response":"The critic is implemented as a zero-shot prompt to the same base LLM used elsewhere in the framework; no fine-tuning occurs. We will include the complete prompt template in the appendix of the revised manuscript. Negative examples are constructed by extracting only the specific failed tactic or lemma name from the verifier output and marking solely that branch as negative for the MCTS value function. The prompt is localized to the current proof state and error type, avoiding global penalties. We will add a brief discussion of potential biases (including lemma-style effects) and the design choices intended to limit them.","revision_made":"yes","referee_comment":"[Experimental setup and critic implementation] No details are provided on the critic's prompt template, whether it is zero-shot or fine-tuned, or how negative-example construction avoids systematic bias across proof states (e.g., over-penalizing viable branches with certain lemma styles); this directly affects the weakest assumption that the critic reliably improves exploration without erasing net gains."}],"tokens_in":1433,"tokens_out":510,"duration_ms":21935,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing here is a two-phase protocol that runs Best-of-N first, then feeds verifier failure signals (syntax errors, type mismatches, partial goals) as explicit negatives into a critic that guides MCTS, while keeping every Phase-1 solve. That setup is new in the cited baselines and produces the reported deltas: 40.6% versus 36.9% Best-of-5 on miniF2F and 7.3% versus 1.8%/3.6% on the released VERITAS-CombiBench. The new benchmark is useful on its own because it shows unguided sampling can hurt when lemma names must be recovered iteratively.\n\nThe paper does a clean job of preserving Phase-1 solves so any extra solves can be attributed to the second phase. Releasing the benchmark and GitHub artifacts is also straightforward and helpful.\n\nThe soft spot is exactly the one the stress-test flags. The whole improvement is credited to the critic turning raw verifier output into useful negative examples without adding bias or wasting search budget. The abstract gives no prompt, no training details, and no ablation that removes the critic while matching compute. Until those are shown, it is hard to know whether the 3.7-point lift on miniF2F or the larger lift on CombiBench actually comes from feedback-driven exploration or from something else in the search setup. The numbers are concrete, but the causal claim is still thin.\n\nThis is for groups working on LLM-guided formal search who already run verifiers and want to test whether richer signals help. It is worth a serious referee because the benchmark is new, the protocol is described clearly enough to replicate, and the empirical comparison is direct. The paper should go to review rather than desk reject.","headline":"VERITAS adds a critic-guided MCTS phase on verifier negatives after Best-of-N sampling, claims 40.6% on miniF2F and 7.3% on a new 55-theorem combinatorics benchmark, but the gains rest on an untested assumption about the critic.","tokens_in":2410,"tokens_out":463,"would_cite":false,"duration_ms":14463,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"VERITAS reaches 40.6% on miniF2F by routing verifier signals into critic-guided MCTS after Best-of-N sampling.","keywords":["formal theorem proving","LLM provers","verifier-guided search","MCTS","miniF2F","zero-shot","proof search","combinatorics benchmark"],"falsifier":"Ablating the negative examples derived from Phase-1 failures and measuring whether the additional solves on miniF2F and VERITAS-CombiBench disappear while Phase-1 solves remain unchanged.","tokens_in":2660,"feed_emoji":"🧮","tokens_out":730,"duration_ms":19081,"temperature":0.7,"pith_summary":"VERITAS shows that LLM formal provers improve when they treat verifier outputs beyond a binary pass/fail by running Best-of-N sampling first then feeding its failures as negative examples into a second-phase MCTS. The protocol guarantees that every theorem solved in the initial sweep remains solved, so any extra successes come from the feedback-driven exploration. On the standard miniF2F benchmark this yields higher success than independent Best-of-5 or Portfolio baselines. The paper also releases a 55-theorem combinatorics set where unguided sampling drops below Portfolio, indicating that iterative lemma recovery from verifier messages requires guided search.","feed_headline":"Verifier feedback lifts theorem proving to 40.6% on miniF2F","feed_subtitle":"Two-phase protocol turns Phase-1 failures into MCTS negatives and adds solves on both miniF2F and a new combinatorics set without losing ear","key_machinery":"The two-phase protocol that ingests every verifier signal as negative examples for critic-guided MCTS after an initial Best-of-N sweep.","core_discovery":"The central claim is that a two-phase zero-shot protocol—Best-of-N sampling followed by a critic that converts Phase-1 verifier failures (syntax errors, type mismatches, partial progress) into explicit negative examples for MCTS—produces additional proofs while preserving all Phase-1 solves, reaching 40.6% on miniF2F and 7.3% on the new VERITAS-CombiBench where Best-of-5 falls to 1.8%.","pith_inferences":["The same two-phase routing of verifier signals could be tested on other interactive theorem provers that already expose detailed error messages.","Releasing the combinatorics benchmark creates a controlled setting for measuring how well different search methods recover lemma sequences over multiple turns.","If the critic can be replaced by a lighter rule-based extractor of failure patterns, the method might run with lower overhead while retaining the reported gains."],"forward_implications":["Theorems solved by the initial Best-of-N sweep are guaranteed to stay solved after the MCTS pass.","Unguided sampling underperforms when the correct lemma names must be recovered iteratively from successive verifier messages.","The released 55-theorem combinatorics benchmark exposes the gap between Best-of-5 and Portfolio methods under feedback-driven conditions.","Rich verifier signals (syntax, types, partial goals) can be used directly as training signals for the critic without additional human annotation."],"fun_headline_variants":["VERITAS reaches 40.6% on miniF2F with two-phase Best-of-N then MCTS","Verifier failures as MCTS negatives yield 7.3% on VERITAS-CombiBench","Protocol preserves Phase-1 solves while reaching 40.6% on miniF2F","Best-of-5 scores 1.8% on VERITAS-CombiBench versus 7.3% with feedback"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"A critic model can reliably turn Phase-1 verifier failure signals into useful negative examples that improve MCTS exploration without introducing bias or inefficiency that erases the gains.","fun_headline_variants_meta":{"raw":{"variants":["VERITAS reaches 40.6% on miniF2F with two-phase Best-of-N then MCTS","Verifier failures as MCTS negatives yield 7.3% on VERITAS-CombiBench","Protocol preserves Phase-1 solves while reaching 40.6% on miniF2F","Best-of-5 scores 1.8% on VERITAS-CombiBench versus 7.3% with feedback"]},"model":"grok-4.3","cost_usd":0.007061,"raw_usage":{"total_tokens":3270,"prompt_tokens":676,"num_sources_used":0,"completion_tokens":106,"cost_in_usd_ticks":70612000,"prompt_tokens_details":{"text_tokens":676,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2488,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":676,"tokens_out":106,"duration_ms":17095,"temperature":1.0,"reasoning_tokens":2488,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T21:36:02.305028+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Ablating the negative examples derived from Phase-1 failures and measuring whether the additional solves on miniF2F and VERITAS-CombiBench disappear while Phase-1 solves remain unchanged.","supporting_citations":[],"review_version":1}