{"id":"0645997a-5158-4844-b0fc-812badd496a8","arxiv_id":"2505.08492","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A 1.5-billion-parameter local LLM fine-tuned on generated PDDL problems achieves 66.1% single-domain and 70.6% multi-domain valid-plan rates, showing lightweight neurosymbolic planners are feasible.","lead":"This paper introduces Gideon, a system that fine-tunes a small open-source language model (Qwen 2.5, 1.5B parameters) to generate robot task plans written in PDDL. It reports valid-plan rates of 66.1% in single-domain tests and 70.6% in multi-domain tests, pointing toward on-device planning that does not depend on cloud services.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validity is measured on DPGC-generated test problems drawn from the same configuration as training, so it may reflect distribution imitation rather than general planning; test on out-of-distribution and minimal instances before accepting the scalability claim.","rationale":"The reader's weakest assumption is essentially the same load-bearing concern I identify: the DPGC-generated data may not be representative of real planning difficulty, so validity measured on held-out generated problems need not transfer to real human-robot collaboration scenarios. My read agrees with that concern and adds a concrete internal red flag from Table 1: Gideon models never generate plans shorter than 15 actions, whereas Probe solves trivial instances in 2 actions, indicating the model has learned distribution-specific output patterns rather than robust planning. The paper deserves credit for its open code, VAL-based validation, duplicate filtering, and explicit limitation statements in Section 5, and the feasibility result is plausible. However, the central numbers do not yet isolate planning generalization from distribution imitation. The proposed out-of-distribution test would settle this: if Gideon performs comparably on benchmark and minimal instances, the concern would be resolved; if not, the conclusions would need to be substantially narrowed. Since the reader already issued a CONDITIONAL verdict and my concern supports that conditionality rather than overturning it, I recommend no change to the reader's verdict.","tokens_in":12940,"tokens_out":4796,"duration_ms":51290,"concrete_test":"Curate an out-of-distribution test set for NO-MACRO and MACRO using sources not produced by Gideon's DPGC: the original human-robot collaboration benchmark instances from Capitanelli et al. [4], hand-written minimal instances with 2-5 action solutions, and instances generated by an independent sampler with different object pools and mutex rules. Run Gideon-NO-MACRO-32k and Gideon-MD-16k on this set and compare VAL-validity with 66.1% and 70.6%. If validity remains comparable, the distribution concern does not land; if it drops substantially or collapses on minimal instances, the paper's claims must be restricted to the generated distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim rests on 66.1% single-domain and 70.6% multi-domain validity measured on held-out problems generated by the same DPGC configuration used to create training data. Section 3.3 asserts that 'the planner cannot distinguish between generated and real-world data,' but this is a conceptual assertion: DPGC encodes user-specified probabilities, object pools, mutexes, and invariants, and the paper explicitly leaves the full format description out of scope. Representativeness of real human-robot collaboration difficulty is therefore an input assumption, not a demonstrated property. Table 1 gives concrete evidence that the trained models are distribution-specific: Gideon plans never fall below 15 actions, while Probe solves minimal instances in as few as 2 actions. A model that cannot recognize trivially solvable problems has not acquired general planning competence; it has learned the output statistics of the generated distribution. If the test problems are structurally similar to training problems, high validity can coexist with poor transfer to genuinely new problem shapes. Thus the headline numbers do not yet establish scalable multi-domain planning outside the generated distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Gideon, a framework for neurosymbolic PDDL planning using locally deployed lightweight LLMs. It contributes a DPGC-based problem generator that produces domain-problem-plan tuples for PDDL 2.1 domains, a pipeline that fine-tunes Qwen-2.5 1.5B on Alpaca-formatted tuples, and evaluations on two articulated-object manipulation domains (MACRO and NO-MACRO). Single-domain models fine-tuned on 8k, 16k, and 32k samples achieve 39.9%, 38.0%, and 66.1% VAL-valid plans on a held-out 1,000-problem test set; a multi-domain 16k model reaches 70.6% validity on a mixed test set and 83.0% on the NO-MACRO subset. The authors position these results as evidence that lightweight local LLMs can support scalable multi-domain planning, using Teriyaki (GPT-3) and Probe as baselines.","tokens_in":13275,"tokens_out":5623,"duration_ms":55297,"significance":"If the transfer assumption holds, the paper's main contribution is a practical, open pipeline for generating large PDDL training corpora and evidence that a 1.5B-parameter model can be fine-tuned to produce mostly valid plans, roughly 120x smaller than the GPT-3 baseline. Strengths include the public code and data, external VAL validation, held-out and deduplicated test sets, and honest acknowledgment that comparison with Teriyaki is against published numbers. The significance is currently limited by the fact that all test problems are generated from the same DPGC configuration as training, and by results being point estimates from single runs; the broader human-robot collaboration scalability claim therefore remains conditional.","major_comments":[{"comment":"The central transfer claim is not yet empirically supported. Section 3.3 asserts that 'the planner cannot distinguish between generated and real-world data,' but this is a representational statement, not evidence about difficulty distribution. All evaluation problems are sampled from the same DPGC configuration used for training. Table 1 already contradicts full generality: Gideon plans never have fewer than 15 steps, while Probe solves trivial instances in 2 steps. To support the title's 'Scalable Robot Autonomy' and the human-robot collaboration motivation, the authors should evaluate on out-of-distribution problems such as different object counts, different mutex and invariant configurations, IPC-style instances, or hand-designed minimal problems, and report validity there. Without such tests, the headline 66.1% and 70.6% figures may reflect imitation of the generated distribution rather than planning competence.","section":"Section 3.3 / Section 4.1 (Tables 1-2)"},{"comment":"All validity numbers are point estimates from a single fine-tuned model and a single test set. No confidence intervals, multiple seeds, or significance tests are reported. The scaling claim is weakened by the non-monotonic sequence 39.9% (8k), 38.0% (16k), 66.1% (32k); the 32k improvement rests on one run. Please provide at least three seeds per configuration, or bootstrap confidence intervals on plan validity, and a test of whether the observed differences are significant given the 1,000-sample test set.","section":"Section 4.1 (Figure 2, Table 1)"},{"comment":"The claimed benefit of multi-domain training is confounded. Gideon-MD-16k is trained on 8,000 NO-MACRO and 8,000 MACRO samples, but no single-domain MACRO model is trained. Its 83.0% NO-MACRO validity is compared against single-domain NO-MACRO models, which does not isolate the effect of 'regularization by data variety' from the effect of including MACRO data or from domain-specific difficulty. Without a single-domain 16k MACRO baseline and a matched single-domain NO-MACRO run, the multi-domain improvement cannot be attributed to data variety; please either add the missing baseline or explicitly frame the multi-domain result as a proof-of-concept without causal attribution.","section":"Section 4.2 (Table 3)"},{"comment":"The DPGC format is a central contribution, but the paper explicitly leaves 'an overly detailed description' out of scope and only sketches probabilities, pools, mutexes, and tagged references. Because the full schema is only in a repository, the exact difficulty distribution of the generated problems cannot be assessed from the manuscript. At a minimum, include the DPGC files used for MACRO and NO-MACRO as supplementary material in the paper, or provide a complete specification in an appendix, so that reviewers and readers can reproduce the data distribution and judge its representativeness.","section":"Section 3.3"}],"minor_comments":[{"comment":"In the paragraph beginning 'Neurosymbolic approaches are an alternative class,' the phrase 'the promise to to address' contains a duplicated 'to'.","section":"Section 2"},{"comment":"The caption refers to 'DPCG file' while the text consistently uses 'DPGC'; please unify the acronym.","section":"Section 3.2 (Figure 1 caption)"},{"comment":"Decimal commas are used inconsistently (e.g., '70,6', '39,9' versus '70.6', '39.9' in the text); please use a single convention throughout.","section":"Section 4.2 (Tables 3 and 4)"},{"comment":"The statement that 66.1% represents a '75% increase' over the 16k model is slightly off; 66.1/38.0 is approximately 1.74. Please write 'approximately 74%' or provide the exact computation.","section":"Section 4.1"},{"comment":"The introduction promises a Section V titled 'Limitations,' but no such section exists; limitations appear only as a brief paragraph in the Conclusion. Please add a dedicated limitations section or correct the roadmap.","section":"Section 5 / overall structure"},{"comment":"The hardware description lists an NVIDIA RTX A6000, yet average planning times exceed 60 seconds for a 1.5B model; please state explicitly whether inference ran on GPU or CPU and whether any quantization was used, as this affects the timing interpretation.","section":"Section 3.4 / Section 4.1 (Table 2)"}],"recommendation":"major_revision","confidential_remarks":"Editors may wish to consider whether the title and framing overpromise relative to the evidence: all empirical support is in-distribution generated problems. The contribution is useful as a systems paper, but the 'Scalable Robot Autonomy' claim needs either re-scoping or out-of-distribution evaluation. Note also the self-referential inconsistency: the paper promises a Limitations section that does not exist, and the abstract claims a modular pipeline 'for any domain' while the empirical support covers only two closely related domains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper is worth a serious look, not because the headline numbers are impressive (they aren't, compared to Teriyaki or Probe), but because the DPGC problem generator for arbitrary PDDL domains is a genuinely useful contribution. The empirical claim — that a 1.5B local model can be fine-tuned to produce valid PDDL plans at 66% on held-out generated problems — is plausible and honestly presented, but it is a feasibility result, not a demonstration of scalable multi-domain autonomy.\n\nWhat is actually new and good: the DPGC format and pipeline. As far as I can tell, it does fill a real gap: a configurable generator that respects domain invariants, handles mutexes and tagged object pools, and can produce large datasets for arbitrary PDDL 2.1 domains. Code and data are public. The authors also deserve credit for not overselling against baselines; they explicitly say they are not trying to beat Teriyaki or Probe, and they cite Teriyaki's published numbers because GPT-3 is unavailable. That comparison is fair, not circular.\n\nThe soft spots are real but addressable. The biggest one is the one the stress-test flags: validity is measured on test problems generated from the same DPGC configuration as training. That means the model may be imitating the distribution of the generator rather than learning general planning competence. The evidence is in Table 1: Gideon's minimum plan length is never below 15 actions, while Probe solves trivial instances in 2. A model that can't recognize an easy problem has not acquired a general planning ability; it has learned the output statistics of the training distribution. The paper's claim that \"the planner cannot distinguish between generated and real-world data\" is a conceptual assertion, not something they tested. They need out-of-distribution tests, or at least minimal hand-crafted instances.\n\nOther soft spots: no error bars or repeated runs, so the validity percentages are point estimates; the 16k model performing worse than 8k is explained by \"delayed generalization\" but that's speculative; the multi-domain experiment uses two very similar domains, so the \"multi-domain\" claim is thin. These are not fatal, but they need to be addressed.\n\nWho is this for? People working on neurosymbolic planning, data generation for planning, or on-device LLM deployers. The DPGC tool alone justifies reading the paper.\n\nRecommendation: send it to peer review. It deserves referee time, with the expectation of major revision. The core feasibility claim holds up at a basic level, and the tool is useful. But the authors need to show more evidence of generalization before the scalability claim can be accepted.\n\nBest.","headline":"A credible feasibility study with a genuinely useful problem generator; the planning validity numbers are real but only demonstrate in-distribution imitation, not yet scalable general planning.","tokens_in":13694,"tokens_out":1467,"would_cite":true,"duration_ms":17262,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a 1.5-billion-parameter local language model, fine-tuned on systematically generated PDDL training data, produces formally valid plans in 66.1% of single-domain tests and 70.6% of multi-domain tests.","keywords":["neurosymbolic planning","PDDL","large language models","task planning","domain-problem-plan generation","human-robot collaboration","fine-tuning","local LLM"],"falsifier":"Run the same trained model on hand-written test problems taken from a real human-robot collaboration, or on a previously unseen PDDL domain the DPGC was not configured for, and compare plan validity; if validity falls well below the 66-71% range while generated-problem validity stays high, the generator's difficulty distribution is not representative and the scalability claim does not transfer.","tokens_in":12730,"feed_emoji":"🤖","tokens_out":5813,"duration_ms":55622,"temperature":0.7,"pith_summary":"This paper argues that robot task planning can be pushed onto small language models that run on local hardware, removing dependence on cloud APIs. The Gideon framework generates large volumes of realistic symbolic planning problems from a PDDL domain description, fine-tunes a 1.5-billion-parameter model on the resulting domain-problem-plan triples, and reports that the model produces formally valid plans in 66.1% of single-domain test cases and 70.6% of mixed-domain test cases. The authors frame the result as proof of feasibility and scalability for on-device neurosymbolic planning, not as a claim of superiority over traditional or larger planners.","feed_headline":"Small local AI model writes valid robot plans 66% of the time","feed_subtitle":"A 1.5-billion-parameter model trained on 32k generated examples plans on-device and generalizes across domains.","key_machinery":"The load-bearing mechanism is the Domain-Problem Generation Configuration (DPGC), a JSON schema that controls random generation of solvable PDDL problem instances from a domain, together with the pipeline that converts generated problems into plans with a classical planner, validates them with the VAL validator, and packs them into instruction-tuning examples where the domain is the instruction, the problem is the input, and the plan is the output. This machinery is what makes large-scale multi-domain training data cheap and unique, and the extended context window of the local model is what allows the full domain and problem to appear together in each example, enabling multi-domain generalization.","core_discovery":"The central claim is that a 1.5-billion-parameter, locally deployable language model can learn to produce formally valid PDDL plans when fine-tuned on large quantities of systematically generated domain-problem-plan triples. Concretely, the paper reports 66.1% valid plans for the best single-domain model trained on 32,000 samples, and 70.6% valid plans for a multi-domain model trained on 16,000 samples from two related articulated-object manipulation domains, with 83.0% validity on one domain alone. The framework is not claimed to beat traditional planners or larger models on speed or sample efficiency; rather, the paper claims it demonstrates feasibility and scalability of on-device neurosymbolic planning with a model about 120 times smaller than the cloud-hosted baseline.","pith_inferences":["Because the multi-domain model beat every single-domain model on NO-MACRO despite seeing only half as much NO-MACRO data, data variety may be a cheaper scaling axis than data volume; testing on domains that share no actions or predicates would show whether the effect is general.","The DPGC's probability and mutex knobs make difficulty an input parameter, so Gideon could be used to study curriculum learning for planning: training on easy problems first and increasing plan length or mutex density may cut the data needed to reach valid planning.","The reported validity measures formal plan correctness, not interaction fluency; an end-to-end test with a real robot and a human would show whether faster first-action availability actually improves human-robot collaboration."],"forward_implications":["A model roughly 120 times smaller than the older cloud baseline can produce valid plans on this class of articulated-object manipulation tasks, making on-device planning practical where network access is unreliable.","Because validity jumped from 38.0% at 16k samples to 66.1% at 32k samples, scaling the generated dataset further should continue to raise the validity ceiling.","Multi-domain training appears to act as a regularizer: the 16k multi-domain model reached 83.0% on the single NO-MACRO test set, higher than any single-domain model, so generalist planning models may be easier to train than specialists.","Including the full PDDL domain in the context, which was not possible with older 2048-token models, is what enables the multi-domain setup, and the same mechanism should extend to more domains as context windows grow.","An action-by-action generation strategy would cut the 32k model's estimated planning time to around 1.6 seconds, beating the traditional planner on responsiveness."],"supporting_citations":[{"why":"Supplies the articulated-object manipulation domains and problem formulation used as the benchmark.","marker":"[4]"},{"why":"Provides the neurosymbolic fine-tuning approach and the published baseline validity numbers Gideon compares against.","marker":"[5]"},{"why":"The validator used to confirm that generated plans and model-produced plans are formally correct.","marker":"[18]"},{"why":"The satisficing planner used to produce ground-truth plans for the training dataset.","marker":"[21]"},{"why":"The compact open model family that Gideon fine-tunes; its extended context window makes domain-plus-problem inputs feasible.","marker":"[30]"}],"fun_headline_variants":["On-device 1.5B LLM yields 66% valid robot plans","Gideon: local LLM with 70.6% multi-domain planning","120x smaller model still plans robots on-device","Local LLM achieves 66% valid plans without cloud","Tiny model, big plans: 70% robot planning success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the user-configured problem generator produces problems whose difficulty and structure match real robot planning tasks, so the validity measured on generated test sets transfers to actual human-robot collaboration.","fun_headline_variants_meta":{"raw":{"variants":["On-device 1.5B LLM yields 66% valid robot plans","Gideon: local LLM with 70.6% multi-domain planning","120x smaller model still plans robots on-device","Local LLM achieves 66% valid plans without cloud","Tiny model, big plans: 70% robot planning success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2317,"prompt_tokens":1032,"completion_tokens":1285,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":648,"completion_tokens_details":{"reasoning_tokens":1193}},"tokens_in":648,"tokens_out":1285,"duration_ms":12451,"temperature":1.0,"reasoning_tokens":1193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:53:13.135063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trained model on hand-written test problems taken from a real human-robot collaboration, or on a previously unseen PDDL domain the DPGC was not configured for, and compare plan validity; if validity falls well below the 66-71% range while generated-problem validity stays high, the generator's difficulty distribution is not representative and the scalability claim does not transfer.","supporting_citations":[{"cited_title":"Robotics and Autonomous Systems 109, 139–155 (2018)","cited_arxiv_id":null,"evidence_quote":"Supplies the articulated-object manipulation domains and problem formulation used as the benchmark."},{"cited_title":"In: 16th IEEE International Conference on Tools with Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"The validator used to confirm that generated plans and model-produced plans are formally correct."},{"cited_title":"In: Proceedings of the International Conference on Automated Planning and Schedul- ing","cited_arxiv_id":null,"evidence_quote":"The satisficing planner used to produce ground-truth plans for the training dataset."}],"review_version":1}