{"id":"5dc466a4-2b1e-4dbc-9f97-42fdaabf000e","arxiv_id":"2507.08501","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Lang2Logic formalizes a problem into a five-field model (type, variables, constraints, objectives) and then generates code to solve it, and when trained with a bilevel GRPO-style objective it beats CoT, Plan-and-Solve, Self-Refine and PAL on nine benchmarks.","lead":"This paper introduces Lang2Logic, a two-stage framework where one language model turns a natural-language problem into a structured formal model and a second model generates executable Python code from that model. The authors report accuracy gains over chain-of-thought and program-aided baselines on nine reasoning benchmarks, with an eye toward interpretable and trustworthy LLM reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may be a training-data artifact: Lang2Logic is SFT/RL-trained on 28K math examples plus GSM8K, while all baselines are untrained 1–3 shot prompts; no matched fine-tuned PAL/CoT baseline isolates the two-stage/bilevel contribution.","rationale":"I focused on the causal attribution in the strongest claim. The reader's weakest_assumption is about OGF generalization and unmeasured formalization accuracy; that is related but not identical. My concern is upstream: even if the OGF were accurate, the comparison design cannot establish that the framework's components cause the gains, because the treatment group receives supervised fine-tuning and RL while control baselines are frozen few-shot models. This is a standard matched-baseline problem, not an internal inconsistency. The paper's own text supports the concern: Section 3.3 constructs a 28K dataset, Section 3.4 describes 5 RL iterations on GSM8K, and Section 4 states baselines are evaluated with 1-3 shots and no training. The absence of appendices/code/error bars and the MoT vs Lang2Logic / relative-vs-absolute / 9-vs-10-benchmark inconsistencies all make the evidence harder to audit, but the matched-training gap is the single load-bearing issue because it determines whether the architecture has any demonstrated effect. A matched-ablation experiment is cheap in principle and would settle it. I therefore keep the reader's CONDITIONAL verdict: the direction is plausible, but the central attribution should not be accepted until matched-baseline and intermediate-formalization evidence is provided.","tokens_in":12084,"tokens_out":8219,"duration_ms":96981,"concrete_test":"Run a matched-training ablation: fine-tune PAL (and CoT) on the identical 28K model-augmented Dmod and apply the same 5-iteration GRPO-style RL on GSM8K with the same qwen2.5 backbones and compute budget, then evaluate on the same benchmarks with the same few-shot answer formats. If Lang2Logic's average gain over this matched PAL baseline remains >10%, the two-stage formalization is supported; if the gap collapses, the gains are attributable to the SFT/RL training corpus rather than to the bi-level framework.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central attribution claim is that the two-stage OGF→LG formalize-then-generate design and the bilevel GRPO objective (Eqs. 8–10) cause the reported accuracy gains. That attribution is not testable from the current experiments. Section 3.3 pre-trains the OGF LLM on a 28K model-augmented math dataset (gsm8k/aqua/qed), and Section 3.4 applies bilevel RL on GSM8K, but Section 4 evaluates all baselines (CoT, Plan-and-Solve, Self-Refine, PAL) in a 1–3 shot prompting setting with no fine-tuning. Lang2Logic therefore receives roughly 28K+ of additional training data and 5 RL iterations before the comparison, while PAL—the strongest baseline—does not. A fine-tuned PAL or CoT trained on the same Dmod and GSM8K/RL budget could plausibly close most of the gap, in which case the headline 10%+ gains would be a training-data/compute effect rather than evidence for the bi-level formalization. The paper also never reports the accuracy of the intermediate five-tuple model m, so we cannot tell whether errors originate in OGF abstraction or LG code generation, or whether the RL stage improves the OGF at all. Absent a matched fine-tuned baseline and an intermediate-formalization measurement, the load-bearing claim that the architecture, rather than the training corpus, produces the gains is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Lang2Logic, a two-stage LLM reasoning framework in which an Optimization-Guided Formalization (OGF) LLM first parses a natural-language question into a five-tuple formal model (problem overview, model type, variables, constraints, objective), and a Logic Generation (LG) LLM then converts that model into an executable Python program. The two stages are trained by supervised fine-tuning on a 28K-sample model-augmented math dataset (constructed from Flan subtasks gsm8k, aqua, qed via rejection sampling) followed by a bilevel GRPO-style reinforcement learning procedure (Eqs. 8–10, Algorithm 1) run on GSM8K. The authors evaluate the resulting Qwen2.5-7B and 1.5B systems on nine benchmarks spanning causal, logical, mathematical, spatial, and temporal reasoning and report average gains above 10% absolute accuracy over CoT, Plan-and-Solve, Self-Refine, and PAL baselines, together with claims of interpretability and error traceability.","tokens_in":12424,"tokens_out":6838,"duration_ms":74282,"significance":"If the empirical claims were fully supported, Lang2Logic would be a useful contribution: the explicit separation of abstraction from executable generation is well motivated, the five-tuple schema is a concrete and interpretable intermediate representation, and the unified GRPO-style objective for the two stages is clearly specified. The paper also covers an admirably broad set of reasoning benchmarks and reports results at two model scales. However, the experiments as presented do not isolate the causes of the reported gains: there is no matched fine-tuned baseline, no ablation of the components (prompted two-stage vs SFT-only vs bilevel RL), no repeated-seed variance reporting, and no evaluation of the intermediate formal model. The central attribution claim is therefore conditional on substantial additional experiments.","major_comments":[{"comment":"The comparison is asymmetric: Lang2Logic receives SFT on 28K model-augmented examples and five iterations of RL on GSM8K, while CoT, Plan-and-Solve, Self-Refine, and PAL are deployed as untrained 1–3 shot prompts. The reported 10%+ average accuracy gains could therefore reflect additional training data and compute rather than the two-stage architecture or the bilevel objective. The central claim requires matched baselines: fine-tuned PAL and CoT on the same Dmod/GSM8K data and the same RL budget, plus a fine-tuned single-stage code-generation model as a control.","section":"Section 4, 'Base Models, Training Data and Experimental Setup'; Tables 1–2"},{"comment":"No ablation separates the contributions of the cold-start SFT (Eq. 7), the bilevel RL (Eqs. 8–10), and the two-stage modeling itself. Without (i) Lang2Logic used in a zero-shot/two-stage prompting mode, (ii) Lang2Logic with SFT only, and (iii) Lang2Logic with full bilevel RL, the claim that the bi-level design causes the gains is not testable. In addition, none of the tables include error bars or multiple seeds, so the significance of small absolute differences (e.g., 3.1% on SVAMP) is unknown.","section":"Section 3.4, Eqs. (8)–(10); Tables 1–2"},{"comment":"The paper says all evaluation benchmarks are tested 'without any training' and that GSM8K is the only in-domain dataset. However, GSM-Hard is described as a 'harder version of GSM8K', and SVAMP is a math word-problem benchmark closely related to the gsm8k/aqua/qed SFT seed corpus. The math-column gains (e.g., +14.4 on GSM-Hard) may therefore be a distribution-shift artifact rather than evidence of cross-domain generalization. The authors should report results on a held-out non-math domain and, ideally, a math benchmark not used in any training stage.","section":"Section 4, 'Evaluation Datasets'; Table 1"},{"comment":"The manuscript states that the OGF–LG interaction 'is inherently bidirectional, supporting iterative refinement', with the LG able to 'regenerate the program or giving feedback to the OGF', but neither Algorithm 1 nor the experiments implement this runtime feedback. The introduction and conclusion nevertheless describe the framework as jointly optimizing and dynamically coordinating the two stages. This mismatch should be resolved by either implementing and ablating the feedback loop or deleting the claim.","section":"Section 3.2, 'Collaborative Framework'; Algorithm 1"},{"comment":"The paper never measures the accuracy or validity of the intermediate five-tuple model m. It therefore cannot support the claimed 'transparency and error traceability', nor can it attribute errors to OGF abstraction versus LG code generation. The authors should report a formalization-level evaluation (e.g., whether m is well-formed and factually faithful, ideally checked by humans or a verifier) and analyze how the bilevel RL stage changes m.","section":"Section 4.1 and Eq. (4)"},{"comment":"The text refers to 'Additional results using in-domain training data are provided in the Appendix C' and to Appendix A/B for dataset and training details, but the manuscript contains no appendices. The missing in-domain training results are directly relevant to the training-asymmetry concern raised above, so this is not a purely cosmetic omission.","section":"Appendices (referenced but missing)"}],"minor_comments":[{"comment":"'We implement MoT on two model scales' should read 'Lang2Logic'; the abbreviation MoT is otherwise unused and appears to be a carryover.","section":"Section 4, first paragraph"},{"comment":"The text mentions 'PoT' and 'planning and solving' as baselines, but Table 1/2 list PAL and Plan-and-Solve; the naming should be consistent with the tables.","section":"Section 4.1, main results text"},{"comment":"The Plan-and-Solve row uses a comma as decimal separator ('54,4'); use '54.4' for consistency.","section":"Table 2"},{"comment":"The paper mixes 'over 10% absolute accuracy gains' with 'gains reaching as high as 40%'; the latter appears to be a relative gain (Table 2, K&K puzzles, 1.5B), and the two statistics should be explicitly distinguished.","section":"Abstract and Section 1"},{"comment":"The description of tokens mixes 'model sequence', 'solving logic sequence', and 'executable program sequence'; please clarify which sequence each factor corresponds to.","section":"Eq. (3)"},{"comment":"The phrase 'cold-start fine-tuning ... before after joint bilevel training' contains a typo ('before after'); 'model-augumented' and 'can be find' should also be corrected.","section":"Section 3.3"},{"comment":"The reference list is severely incomplete: many entries lack authors and titles (e.g., '(2000). An introduction to mathematical modeling.'), preventing verification; please supply full bibliographic information.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The editor may wish to ask the authors to state explicitly whether code, data, and trained checkpoints will be released; the manuscript currently makes no such statement, which is a reproducibility concern. The authors cite several of their own prior bilevel-optimization papers (e.g., Y. Liu, Kai Yang, Keying Yang et al.); the relationship between those works and the proposed bilevel RL formulation should be clarified to avoid overclaiming novelty. Finally, the manuscript header date (September 23, 2025) conflicts with the arXiv v1 date (July 11, 2025); this should be reconciled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core idea here is genuinely worth a look: split reasoning into an OGF stage that emits a five-tuple formal model (p, t, V, C, O) and an LG stage that writes executable Python, then train both with a bilevel GRPO objective. That specific combination — formalize-then-code with joint RL across two LLMs — is not in PAL, PoT, Logic-LM, or the other cited work; the bilevel training of the two policies is the most novel piece. Consistent wins on nine benchmarks at two model scales (7B and 1.5B) are more than you usually see in this subfield, and the out-of-domain spread (causal, temporal, spatial, logic) supports the claim that the recipe generalizes.\n\nBut the experiments as reported do not support the attribution. The stress-test concern is real and I think it lands. The method gets SFT on 28K model-augmented math samples plus 5 rounds of GRPO on GSM8K; every baseline is evaluated 1–3 shot with no training. A fine-tuned PAL or CoT trained on the same data could plausibly close most of a 10% gap, so the headline gain cannot be assigned to the bi-level architecture. The paper also never measures accuracy of the intermediate formal model m, so you cannot tell whether errors come from the OGF or the LG, or whether RL actually improves the OGF. The bidirectional feedback loop in Sec 3.2 (LG regenerating or feeding back to OGF) is described in the architecture but not implemented in the experiments, yet the claims are framed as full-system results.\n\nThere are smaller but real issues: no error bars or repeated seeds; no ablation isolating bilevel RL from plain SFT or two-stage prompting; the intro says \"absolute accuracy gains\" while the tables report \"Relative Δ Gain\"; and there are internal inconsistencies (the experiment section says \"MoT\" where it means Lang2Logic, and one table cell is \"54,4\"). No code, data, or appendices, despite several \"see Appendix\" pointers.\n\nWho this is for: anyone working on LLM reasoning with executable code or on training multi-model pipelines. The architectural recipe is a reasonable road to explore, and the paper deserves a serious referee — but it needs matched fine-tuned baselines, intermediate-formalization measurements, ablations, and corrected reporting before the main claim is credible. I would send it to review, with those additions required before acceptance.","headline":"A plausible formalize-then-code framework whose most novel piece is bilevel GRPO training of two LLMs, but the reported 10%+ gains are confounded by an unmatched training-data advantage and missing ablations.","tokens_in":12948,"tokens_out":2070,"would_cite":false,"duration_ms":26143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lang2Logic claims that splitting LLM reasoning into formalization then executable code yields average absolute accuracy gains above ten points across nine reasoning benchmarks, with the largest gains on complex multi-step problems.","keywords":["structured reasoning","bilevel optimization","formalization","logic generation","program-aided reasoning","GRPO","interpretability"],"falsifier":"Run Lang2Logic on a held-out mix of the nine benchmarks under three conditions: the original OGF five-tuples, expert-written oracle five-tuples, and five-tuples with constraints or variables randomly corrupted. If accuracy does not drop under corruption, formalization is not doing the causal work the paper assigns it; if oracle tuples raise accuracy markedly, formalization errors are the bottleneck the end-to-end numbers conceal.","tokens_in":11854,"feed_emoji":"🧠","tokens_out":7860,"duration_ms":77255,"temperature":0.7,"pith_summary":"Lang2Logic claims that large language models reason more accurately when reasoning is split into two explicit stages: first abstracting the problem into a structured formal model, then translating that model into executable Python. On nine benchmarks spanning causal, logical, mathematical, spatial, and temporal reasoning, the paper reports average absolute accuracy gains above 10 percentage points over the strongest single-stage baselines, with the largest gains on the hardest multi-step problems. The separation is meant to make the reasoning process interpretable and debuggable, because each stage produces a human-readable artifact—a five-tuple problem model and a program—that can be inspected independently. A bilevel reinforcement-learning objective, built on GRPO-style updates, trains the formalization model and the code-generation model jointly, which the paper argues is what lets the two stages cooperate rather than drift apart.","feed_headline":"Formalize then code lifts LLM reasoning by over 10 points","feed_subtitle":"A five-tuple problem model feeds executable Python, beating CoT and PAL on causal, spatial, temporal, and logic tasks.","key_machinery":"The central object is the five-tuple formal model $m = (p, t, V, C, O)$ produced by the OGF LLM: $p$ is a problem overview, $t$ the model type (e.g., probabilistic calculation, SAT, CSP), $V$ the decision variables, $C$ the symbolic constraints, and $O$ the objective. This schema is the interface between the two stages; it carries the burden of disentangling the problem's underlying logic from linguistic noise. The optimization machinery is the bilevel GRPO-style objective of Eqs. (8)--(10), with the lower-level LG policy updated on program and answer rewards and the upper-level OGF policy updated on the averaged rewards of its sampled models, solved by alternating updates. The supporting data machinery is a model-augmented cold-start dataset $D_{mod} = \\{q, a, s, m, p \\mid \\mathrm{Exec}(p) = a, \\mathrm{Rank}(m)\\}$ built by rejection sampling from Flan subtasks.","core_discovery":"The paper's central claim is that the bottleneck in LLM reasoning is not computation but the absence of an explicit modeling step. Its proposed Lang2Logic pipeline uses an upper-level Optimization-Guided Formalization (OGF) LLM to convert a natural-language query into a five-tuple model $m = (p, t, V, C, O)$ (problem overview, model type, variables, constraints, objective), and a lower-level Logic Generation (LG) LLM to turn that model into a symbolic workflow and executable Python code whose output is the answer. The framework is trained in two phases: supervised cold-start on a model-augmented dataset built by rejection sampling and preference self-evaluation, then joint bilevel RL that alternates updates of the lower-level policy and upper-level policy using rule-based accuracy and format rewards with normalized advantages. Evaluated without training on nine out-of-domain benchmarks with qwen2.5-7B and qwen2.5-1.5B backbones, the paper reports consistent improvements over CoT, Plan-and-Solve, Self-Refine, and PAL, including relative gains as high as 39.7% on individual tasks.","pith_inferences":["The paper never measures whether the OGF five-tuple is correct, so an ablation that swaps in oracle or corrupted formal models is needed to determine whether the gains come from formalization or from the code generator; that isolation is not in the paper.","Section 3.2 describes a bidirectional feedback loop, but the experiments evaluate a feed-forward pipeline; implementing the loop could raise accuracy further or introduce instability, and the paper's headline numbers do not test it.","A natural extension is to use the five-tuple as a shared inter-agent representation in multi-agent or tool-use settings, where the formal model would let different modules coordinate on one problem description."],"forward_implications":["The same two-stage pipeline should transfer to new reasoning domains by reusing the five-tuple schema, without per-benchmark retraining, if the reported cross-domain gains are real.","Errors become traceable to either the formalization step or the code step, because both artifacts are human-readable; this supports inspecting and fixing LLM reasoning rather than treating it as a black box.","Smaller models (1.5B) benefit substantially, so structured decomposition may serve as a capacity-efficiency lever for deploying reasoning systems on limited hardware.","Because Python is the unified action space, the same architecture can invoke external solvers and libraries, broadening the framework beyond arithmetic to constraint satisfaction and logical inference."],"supporting_citations":[{"why":"Defines the chain-of-thought baseline that motivates the structured-reasoning critique and serves as the primary comparison paradigm.","marker":"J. Wei et al. 2022"},{"why":"Supplies PAL, the direct program-aided baseline that Lang2Logic must beat, and the code-as-executable-logic idea it extends.","marker":"Gao et al. 2023"},{"why":"Provides the GRPO rule-based RL algorithm whose group-relative advantages and clipping the bilevel objective adapts.","marker":"Shao et al. 2024"},{"why":"Inspires the cold-start supervised fine-tuning and rule-based reward design used in the two-stage training.","marker":"Guo et al. 2025"},{"why":"Supplies the Flan-Collection subtasks (gsm8k, aqua, qed) used to construct the model-augmented SFT dataset.","marker":"Longpre et al. 2023"},{"why":"Provides GSM8K, the training and in-domain benchmark for the cold-start SFT and the bilevel RL stage.","marker":"Cobbe et al. 2021"},{"why":"Defines the qwen2.5-7B and qwen2.5-1.5B backbone models on which all experiments run.","marker":"A. Yang et al. 2024"},{"why":"Provides the code-enhanced rejection sampling used to filter the model-augmented dataset to correct, ranked exemplars.","marker":"Guan et al. 2025"}],"fun_headline_variants":["Two-step language-to-logic lifts LLM reasoning by 40%","Bi-level framework turns language into logic, boosts reasoning","Explicit modeling beats raw compute in LLM reasoning","From prose to Python: bi-level LLM reasoning leap","Divide and conquer: LLM reasoning with language-to-logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the OGF LLM, cold-started on math-flavored tasks, produces correct five-tuple formal models for out-of-domain causal, spatial, temporal, and logic questions; the paper never measures formalization accuracy separately, so any end-to-end gain could in principle come entirely from the code-generation half.","fun_headline_variants_meta":{"raw":{"variants":["Two-step language-to-logic lifts LLM reasoning by 40%","Bi-level framework turns language into logic, boosts reasoning","Explicit modeling beats raw compute in LLM reasoning","From prose to Python: bi-level LLM reasoning leap","Divide and conquer: LLM reasoning with language-to-logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3368,"prompt_tokens":1003,"completion_tokens":2365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2296}},"tokens_in":619,"tokens_out":2365,"duration_ms":18924,"temperature":1.0,"reasoning_tokens":2296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:17:21.123289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Lang2Logic on a held-out mix of the nine benchmarks under three conditions: the original OGF five-tuples, expert-written oracle five-tuples, and five-tuples with constraints or variables randomly corrupted. If accuracy does not drop under corruption, formalization is not doing the causal work the paper assigns it; if oracle tuples raise accuracy markedly, formalization errors are the bottleneck the end-to-end numbers conceal.","supporting_citations":[],"review_version":1}