{"id":"898ff04b-0f6b-406d-9f9e-c1595986c3dd","arxiv_id":"2501.00830","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM+AL, an LLM-to-BC+ pipeline with solver-feedback self-revision, outperforms direct LLMs and LLM-generated Python on complex action-reasoning puzzles, though it still needs a few manual corrections on most hard cases.","lead":"This paper builds a pipeline that uses an LLM to translate natural-language action puzzles into the action language BC+, then uses the BC+ reasoner to find plans and give feedback for revision. On Missionaries and Cannibals elaborations and puzzle variants, the pipeline beats direct LLM prompts and LLM-written Python, but still needs a few expert corrections on most hard problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is not yet supported: LLM+AL receives author-administered semantic corrections on failed BC+ programs, while the 'even with human feedback' baseline is a single anecdotal ChatGPT-4 exchange on one problem.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation compares an expert-corrected symbolic pipeline against baselines that receive no comparable human assistance. This concern is not a minor caveat; it is the difference between the paper's headline claim and a more modest claim that LLM+AL plus human modeling effort can solve these puzzles. The appendix actually strengthens the concern, because it makes visible that many 'manual corrections' are semantic additions of exactly the elaboration-specific rules that baselines would need to be told. The paper deserves credit for releasing code, documenting prompts, and enumerating error cases, which makes the proposed test feasible. I am not claiming the result is false; the directional advantage of a declarative representation plus solver feedback is plausible and the self-revision statistics suggest real value. But the evidence as presented does not rule out the simpler explanation that the human corrections, not the action-language bridge, drive the performance gap. The internal inconsistency about ChatGPT-4's solved count further suggests that the reported numbers need independent verification before the strongest comparative claim is accepted. A conditional verdict is therefore appropriate: the method is a credible pipeline contribution, but the headline comparative claim about human feedback and consistent correctness is not yet established. This does not change the reader's conditional verdict.","tokens_in":43663,"tokens_out":4835,"duration_ms":48252,"concrete_test":"Run a matched human-feedback experiment using the public code and prompts. For each of the 30 problems, take the final ChatGPT-4 and ChatGPT-4+Code outputs and have an independent expert, blind to condition, repair them for up to the same number of correction rounds that LLM+AL required (or the same annotation budget), using the same feedback style of pointing out the first concrete error. Record the fraction of problems solved and the number/type of corrections. If the assisted baselines reach comparable accuracy to LLM+AL's corrected 100%, the claim 'standalone LLMs fail to improve even with human feedback' is falsified; if they remain near zero despite the same expert effort, the claim survives. Separately, have two independent BC+ experts re-count LLM+AL's corrections from Appendix D to verify the reported 3.1 and 2.2 averages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that LLM+AL's advantage comes from combining LLMs with action-language reasoning, not from the unequal access to expert correction. The paper's support for this is asymmetric. For LLM+AL, Tables 1-2 report averages of 3.1 and 2.2 manual corrections per unsolved problem, but Appendix D shows these corrections are often substantive modeling decisions, not typo fixes: e.g., MCP #17 needs a missing `hungry` causation rule, MCP #8 needs missing cannibal-outnumbering constraints, and MCP #14 needs a missing `bail` rule. Each such correction encodes the elaboration's semantics, so 'minimal human corrections' conflates small edit counts with large amounts of injected domain knowledge. On the baseline side, the claim that standalone LLMs 'fail to improve even with human feedback' rests on a single undocumented 50-iteration ChatGPT-4 exchange on MCP #13, with no correction counts, no independent annotators, and no human-feedback condition for ChatGPT-4+Code, o1-preview, Claude, or Gemini. Additionally, the evaluation uses one run per problem, and the paper is internally inconsistent about ChatGPT-4's MCP performance: Section 4.1 says it solved only three problems, while Table 1 lists six checkmarks (basic, #2, #10, #11, #14, #19). The most plausible alternative explanation is therefore not excluded: LLM+AL succeeds because a human expert supplies omitted domain knowledge in a high-level declarative language, while the baselines are denied equivalent expert assistance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LLM+AL, a neuro-symbolic pipeline that uses an LLM (o1-preview) to generate programs in the action language BC+ from natural-language problem descriptions, then iteratively revises those programs using feedback from the Cplus2ASP solver. The method is evaluated on McCarthy's Missionaries-and-Cannibals elaborations and on several other puzzle domains, with comparison baselines including ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, o1-preview, and ChatGPT-4+Code. The authors report that LLM+AL solves 7 of 17 MCP problems fully automatically and the remaining ones with an average of 3.1 human corrections, while standalone LLMs and LLM+Code perform worse and, the authors claim, do not improve even with human feedback. The paper also reports that its self-revision stage raises program executability from 22.3% to 86.7% and correct-answer rate from 16.6% to 50% over the 30 tested problems. The appendix provides the full prompts, concrete BC+ programs for the basic MCP, and a detailed enumeration of the 42 post-revision issues categorized into signature, rule, and query problems.","tokens_in":43898,"tokens_out":3538,"duration_ms":33745,"significance":"If the reported advantage is robust, the paper would make a meaningful contribution to neuro-symbolic reasoning about actions: it demonstrates that a declarative action language (BC+) combined with an LLM-based semantic parser and solver feedback can approach expert-level correctness on elaboration-tolerant planning puzzles, and it provides concrete evidence of the types of errors that remain. The authors deserve credit for shipping a reproducible artifact: the repository contains the baseline outputs, the exact prompts, and the BC+ programs, and the appendix's error catalog (Appendix D) names specific missing rules (e.g., the 'hungry' causation in MCP #17, the cannibal-outnumbering constraints in MCP #8, and the 'bail' rule in MCP #14) rather than vague summaries. Because correctness is verified by an external solver against known gold-standard plans, the evaluation is not circular. However, the headline comparison is currently weakened by the asymmetric treatment of human correction: LLM+AL receives author-administered corrections whose counts are reported, while the baselines' 'human feedback' condition is essentially anecdotal.","major_comments":[{"comment":"The claim that standalone LLMs 'fail to improve even with human feedback' is not supported by commensurable evidence. The only human-feedback baseline is one anecdotal 50-iteration ChatGPT-4 exchange on MCP #13, with no protocol, no correction counts, and no human-feedback conditions for Claude, Gemini, o1-preview, or ChatGPT-4+Code. Meanwhile, LLM+AL receives author-administered corrections whose counts (3.1 and 2.2 per unsolved problem) are reported. As Appendix D shows, many of these corrections are substantive modeling decisions (e.g., MCP #17 needs a missing 'hungry' causation rule; MCP #8 needs the full cannibal-outnumbering constraints; MCP #14 needs a missing 'bail' rule), not trivial typo fixes. The paper should either provide a controlled comparison where baselines receive an equivalent amount of expert correction (with the same correction-effort metric), or explicitly narrow the claim to 'LLM+AL with a few manual corrections' without the comparative 'even with human feedback' statement.","section":"§4.2, Tables 1–2"},{"comment":"There is an internal inconsistency in the reported ChatGPT-4 performance on the MCP benchmark. The text says 'ChatGPT-4 solves only three' MCP problems, but Table 1 shows six checkmarks for ChatGPT-4: MCP basic, #2, #10, #11, #14, and #19. This discrepancy affects the credibility of the headline comparison and the interpretation of 'solve.' The authors should clarify the counting criterion (e.g., whether optimality or manual re-checking is required) and ensure the prose and table agree.","section":"§4.1 vs. Table 1"},{"comment":"The experiments use a single run per problem per method, with no reporting of sampling variability for the stochastic LLM components. The abstract and conclusion use strong language such as 'consistently leads to correct answers,' and the self-revision statistics (22.3% to 86.7% executability, 16.6% to 50% correct) are presented as if they were deterministic. Because LLM outputs vary across runs, the observed 7/30 and 15/30 counts could change materially with different seeds or temperatures. The paper should report multiple runs (at least for LLM+AL and preferably for the strongest baselines such as o1-preview) or explicitly state that results are from a single run and weaken 'consistently' accordingly.","section":"§4.1 and Tables 1–2"}],"minor_comments":[{"comment":"The rendering of model names as 'C HATGPT-4', 'C LAUDE 3 O PUS', and 'G EMINI ULTRA 1.0' appears in the abstract, introduction, and tables; this appears to be a spacing artifact and should be fixed.","section":"Throughout"},{"comment":"The 'missing necessary rules' entry for MCP #17 ('A rule which causes hungry to be true is required') is listed twice verbatim. One of the duplicates should be removed.","section":"Appendix D.2"},{"comment":"The paper reports 'manual corrections' counts but never defines what counts as one correction (e.g., one edited rule vs. one logical issue). A brief definition would make the 3.1 and 2.2 averages interpretable.","section":"§4.1 and Appendix D"},{"comment":"The sentence about not guaranteeing non-existence of plans of arbitrary length points to 'Sec 6.6 of (Lee 2005)' with no appendix material; since the thesis is a dissertation, the reference should be expanded or the relevant technique summarized so the reader can verify the argument.","section":"§4.2, footnote 5"},{"comment":"The column header 'Programs Issues' reads awkwardly and the table would benefit from clearer alignment between the subcategories (MD, Syn, Sem) and the 'Signature/Rules/Query' rows; this is purely a presentation matter.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a genuinely interesting pipeline and a useful, concrete error corpus, but the review protocol is currently too asymmetric for the strong comparative claim in the abstract. The internal inconsistency between §4.1 and Table 1 also needs to be resolved. I believe the issues are fixable within the manuscript's scope, and I would encourage the authors to run repeated trials and to either add a controlled human-feedback baseline or soften the claim. If they can do that, the paper could be suitable for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this paper primarily for the pipeline. The idea of using an LLM to produce a BC+ signature, English knowledge, and rules, then letting the Cplus2ASP solver drive self-revision, is a real and useful step beyond LLM-to-PDDL or LLM-to-Python. The appendix is the best part: full BC+ programs, the exact prompts, and a detailed enumeration of 42 residual issues with concrete fixes. That transparency makes the work reproducible and is a credit to the authors.\n\nThe benchmark design is also reasonable. McCarthy's MCP elaborations are a good stress test for elaboration tolerance, and the directional result—LLM+AL solves more problems than direct LLM prompting, and the solver catches many issues automatically—is credible. The self-revision numbers (executability from 22% to 87%, correctness from 17% to 50%) are concrete and useful.\n\nThe soft spots are in the headline comparison. The claim that LLM+AL, with relatively minimal human corrections, beats standalone LLMs is weakened by three things. First, the correction counts are small in number but not in content: Appendix D shows several 'one correction' cases that inject a missing cannibal-outnumber constraint or a whole bail action. That is domain knowledge, not a typo fix. Second, the human-feedback baseline for standalone LLMs is a single undocumented 50-iteration exchange with ChatGPT-4 on one problem. That does not support the general statement that standalone LLMs fail to improve even with human feedback. Third, the paper is internally inconsistent: Section 4.1 says ChatGPT-4 solved only three MCP problems, while Table 1 lists six checkmarks. The authors need to fix that and add a protocol for author corrections, ideally with an independent annotator.\n\nNone of this kills the paper. The central pipeline contribution is real, and the ecosystem—code, prompts, detailed error logs—is exactly what a good systems paper should ship. But the evaluation should be qualified: the present evidence supports 'LLM+AL is a promising approach requiring human help,' not 'LLM+AL consistently outperforms LLMs with comparable human feedback.' A referee should ask for careful revision: more runs, a proper human-feedback condition for all baselines, and a clearer separation between automatic self-revision and author interventions.\n\nThis is for researchers working on neurosymbolic planning and action reasoning. It deserves a serious referee, and with revisions it could be a solid contribution. I would send it to review, with the caveat that the comparative claims need to be rewritten to match the evidence.","headline":"Promising LLM-to-BC+ pipeline with a transparent appendix, but the headline comparison overstates the evidence: human corrections are often domain-knowledge injections, and the human-feedback baseline is a single anecdote.","tokens_in":44482,"tokens_out":2470,"would_cite":true,"duration_ms":21620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairing a large language model with the action language BC+ can solve complex action-reasoning puzzles that standalone LLMs miss, the paper claims, with only a few manual corrections to the generated logic program.","keywords":["large language models","action languages","BC+","automated planning","semantic parsing","self-revision","elaboration tolerance","neuro-symbolic reasoning"],"falsifier":"A controlled repair study would settle it: take the 30 problems, give an independent BC+ expert the same correction budget for LLM+AL's outputs, and give an equally skilled expert the same budget to repair ChatGPT-4, o1-preview, and ChatGPT-4+Code outputs, including fixing Python plans directly; if any baseline reaches LLM+AL's correctness at comparable effort, the paper's main claim fails. A quicker check is to repeat the reported 50-feedback ChatGPT-4 exchange on MCP elaboration #13 with a fixed protocol and see whether pointing out constraint violations ever converges to a correct plan.","tokens_in":43401,"feed_emoji":"🧩","tokens_out":9189,"duration_ms":80029,"temperature":0.7,"pith_summary":"The paper tries to establish that current LLMs, though strong at natural language, cannot reliably do action reasoning that requires systematic search, but that this weakness can be bridged by having the LLM generate descriptions in the action language BC+ and letting a symbolic reasoner find plans. Using McCarthy's Missionaries-and-Cannibals elaborations and puzzle variations, it reports that the LLM+AL pipeline consistently reaches correct answers with only a few human corrections, while standalone ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, o1-preview, and ChatGPT-4 with a code interpreter largely fail and do not improve with human feedback. If this holds, it matters because it identifies a practical division of labour: LLMs supply commonsense knowledge and semantic parsing, while declarative action languages supply the search and state-constraint enforcement that LLMs lack. It would also make action-language programming more accessible by automating much of the translation from English to formal rules.","feed_headline":"LLM+AL reliably solves action puzzles with only a few human fixes","feed_subtitle":"It translates English into BC+ logic rules, lets a symbolic reasoner search, and repairs errors from solver feedback.","key_machinery":"The load-bearing object is the action language BC+, a formal language whose declarative causal laws (causes, nonexecutable, impossible, increments, and decrements) describe transitions between states, with support for indirect effects, defaults, and additive fluents. The Cplus2ASP reasoner turns BC+ descriptions into answer-set programs and supplies the search and constraint solving. The pipeline's mechanism is a four-stage loop: the LLM generates a BC+ signature, extracts English knowledge, translates it into BC+ rules and a query, then self-revises using satisfiability checks and sample-query feedback from the reasoner, with a human correcting only the residual errors.","core_discovery":"The paper's central claim is that an LLM can be made reliable at complex action reasoning by using it not as a final reasoner but as a generator of BC+ action-language descriptions, with a symbolic reasoner doing the search and a self-revision loop that feeds solver output back to the LLM. On McCarthy's elaboration-tolerant Missionaries-and-Cannibals benchmark and additional puzzle variations, LLM+AL automatically solves a substantial subset and reaches correct plans on the rest with an average of 3.1 (MCP) and 2.2 (other puzzles) manual corrections to the generated BC+ program. The paper contrasts this with ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, o1-preview, and ChatGPT-4 with a code interpreter, which either produce wrong plans or cannot be repaired by human feedback.","pith_inferences":["A testable extension the paper does not run: apply the same pipeline to BC+ domains it mentions but does not benchmark, such as temporal prediction and postdiction, and measure whether correction counts stay low when the action vocabulary is larger.","A critical check the paper does not include: give independent annotators the same correction budget to repair ChatGPT-4+Code's Python plans directly on the same 30 problems, and compare accuracy at matched effort.","If the reported 50-iteration ChatGPT-4 failure is typical, the lesson generalizes beyond this benchmark: LLM reasoning without an external state-validating oracle cannot be repaired by pointing at mistakes, so methods that add a formal checker may be the only path to reliable correction."],"forward_implications":["LLM+AL would solve 7 of 17 Missionaries-and-Cannibals elaborations outright, with an average of 3.1 manual corrections to the generated BC+ program for the rest, and on 13 additional puzzle and variation problems it solves 8 outright with an average of 2.2 corrections for the rest.","The self-revision stage would carry most of the repair burden: before it, only 22.3% of generated programs are executable and 16.6% produce correct answers, while after solver feedback executability rises to 86.7% and correct answers to 50%.","Standalone ChatGPT-4, Claude 3 Opus, Gemini Ultra 1.0, o1-preview, and ChatGPT-4 with a code interpreter would remain unable to produce consistently correct plans on these problems, and human feedback pointing out errors would not fix the direct-LLM failures.","Because BC+ is declarative, the pipeline would not need to encode a search strategy; the Cplus2ASP reasoner supplies search and constraint satisfaction, so correctness no longer depends on the LLM's ability to write correct search code.","If the pipeline's translation quality holds, action-language descriptions for new domains could be generated automatically rather than hand-authored by experts."],"supporting_citations":[{"why":"Defines the Missionaries-and-Cannibals elaborations that serve as the paper's main benchmark for elaboration tolerance.","marker":"(McCarthy 1998)"},{"why":"Introduces the action language BC+ whose signature, causal laws, and additive fluents the pipeline must generate.","marker":"(Babb and Lee 2015, 2020)"},{"why":"Presents the Cplus2ASP reasoner that checks satisfiability, answers sample queries, and gives the feedback used in self-revision.","marker":"(Babb and Lee 2013)"},{"why":"Provides the causal-theory semantics of action language C+ that underlies BC+ and supports the argument that PDDL is less expressive.","marker":"(Giunchiglia et al. 2004)"},{"why":"Closest prior approach that uses an LLM to generate PDDL and reports that many expert corrections are needed, motivating the BC+ pipeline.","marker":"(Guan et al. 2023)"},{"why":"Documents that LLMs fail on planning benchmarks, providing the baseline context the paper compares against.","marker":"(Valmeekam et al. 2023)"}],"fun_headline_variants":["LLM+AL: LLM writes logic, reasoner solves, needs ~3 fixes","Action reasoning: LLM+AL fixes it with ~3 human tweaks","LLM+AL: ~3 fixes is all it takes for action puzzles","LLM+AL: One LLM to translate, one reasoner to solve, few fixes","LLM+AL: Convert language to logic, then search, with ~3 fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes that the authors' handful of manual corrections to BC+ programs is genuinely less effort than the human feedback that standalone LLMs were given, even though the paper only documents one uncontrolled 50-iteration ChatGPT-4 exchange and does not count correction effort for the baselines.","fun_headline_variants_meta":{"raw":{"variants":["LLM+AL: LLM writes logic, reasoner solves, needs ~3 fixes","Action reasoning: LLM+AL fixes it with ~3 human tweaks","LLM+AL: ~3 fixes is all it takes for action puzzles","LLM+AL: One LLM to translate, one reasoner to solve, few fixes","LLM+AL: Convert language to logic, then search, with ~3 fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001754,"raw_usage":{"total_tokens":6895,"prompt_tokens":887,"completion_tokens":6008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":5898}},"tokens_in":503,"tokens_out":6008,"duration_ms":42479,"temperature":1.0,"reasoning_tokens":5898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:41:19.317851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled repair study would settle it: take the 30 problems, give an independent BC+ expert the same correction budget for LLM+AL's outputs, and give an equally skilled expert the same budget to repair ChatGPT-4, o1-preview, and ChatGPT-4+Code outputs, including fixing Python plans directly; if any baseline reaches LLM+AL's correctness at comparable effort, the paper's main claim fails. A quicker check is to repeat the reported 50-feedback ChatGPT-4 exchange on MCP elaboration #13 with a fixed protocol and see whether pointing out constraint violations ever converges to a correct plan.","supporting_citations":[],"review_version":1}