{"id":"63d262c7-3b8c-425d-8c17-69806027edd7","arxiv_id":"2608.07955","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"NeSy-Spatial self-evolving skills improve spatial reasoning accuracy on MMSI, MindCube, and OmniSpatial, with higher first-pass tool-chain success than prior tool-augmented agents.","lead":"NeSy-Spatial is a new framework that lets spatial-reasoning AI agents convert past tool calls and geometric computations into reusable skills, then apply those skills to later questions. It matters because it is an attempt to make vision-language models more reliable at fine-grained spatial tasks, which is important for robotics, autonomous driving, and embodied AI.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim rests on a single run; on GPT-5.4 the margins over the best baseline are 1.50, 0.49, and -0.73 points, so run-to-run variance could erase the claimed improvement.","rationale":"I read the paper as an engineering claim: a self-evolving neuro-symbolic skill library improves spatial reasoning accuracy and tool success over existing agents. The mechanism is plausible and the controlled Frozen ablation is the right kind of evidence for the evolution component; the case studies also make the skill-update loop concrete. My concern is not with the mechanism but with the evidence for the headline empirical claim. The smallest margins (GPT-5.4 MMSI +1.50, MindCube +0.49) are smaller than typical run-to-run variation for GPT-class APIs in tool-use benchmarks, and there is no statement of how many trials were run, what temperature was used, or how the online stream order was chosen. Prequential accuracy over a single stream conflates learning-curve effects with steady-state performance, so an unlucky order can change the endpoint by more than the reported margin. The absence of error bars is not cured by the fact that results appear consistent across six settings, because all six settings share the same protocol and would share the same variability source. A multi-seed rerun is therefore the one check that would settle the central claim. If the margins are stable, the CONDITIONAL verdict can become ACCEPT; if they are not, the claim should be weakened to a qualitative demonstration of the mechanism. I agree with the reader's weakest-assumption identification and recommend keeping the verdict unchanged until such evidence is provided.","tokens_in":24471,"tokens_out":6655,"duration_ms":74164,"concrete_test":"Run NeSy-Spatial and the strongest baseline on each of the three datasets 10 times, using at least two different online stream orderings and a nonzero sampling temperature (e.g., 0.2), and report mean accuracy, standard deviation, and a paired bootstrap confidence interval for the difference on each backbone. If the 95% CI for any reported GPT-5.4 margin (1.50, 0.49, -0.73) includes zero, the 'consistently improves' claim is not supported. Report the same variability measures for FTCS, Python success rate, and the Frozen ablation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that NeSy-Spatial 'consistently improves reasoning accuracy with more precise tool utilization,' is supported only by one pass through each benchmark. Table 1 and Figure 5 report point estimates with no standard deviation, no multiple seeds, and no significance tests, and the text never states the sampling temperature or number of trials. The decisive GPT-5.4 margins are small: +1.50 on MMSI, +0.49 on MindCube, and -0.73 on OmniSpatial relative to the strongest baseline. On Gemini the gains are larger, but one run cannot distinguish a real effect from API stochasticity or from the particular random order of the online stream. The same dependence affects the FTCS and Python success rates and the Frozen comparison in Figure 6. Because the main result is an empirical claim, the absence of any variability estimate is a load-bearing gap, not a cosmetic one. Appendix D also states that full prompts 'will be available after the code release,' but no code or release link is provided, so the exact prompts and LLM-generated skills cannot yet be inspected.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NeSy-Spatial, a neuro-symbolic framework for online tool-augmented spatial reasoning. The system represents spatial procedures as typed atomic instructions, composes them into Tool-Use Skills for organizing tool calls and Geometry Skills for structured geometric computation, and maintains a skill library that is retrieved and executed in a closed loop during inference. After buffered trajectories are accumulated, the library is updated through LLM-based trajectory analysis, skill fusion, and pruning. The authors evaluate on MMSI, MindCube, and OmniSpatial with GPT-5.4 and Gemini 2.5 Pro, reporting prequential accuracy, first-pass compositional tool-chain success (FTCS), Python success rate, a Frozen ablation, and a component ablation. The central claim is that NeSy-Spatial consistently improves spatial reasoning accuracy and tool-utilization precision, achieving the best aggregate accuracy in five of six backbone-dataset settings.","tokens_in":24748,"tokens_out":6642,"duration_ms":73554,"significance":"The framework addresses a genuine gap: combining tool planning with reusable, verifiable geometric procedures in spatial reasoning. The test-then-update protocol is a real methodological strength, because each prediction is recorded before its label is used for evolution, and the Frozen control in Figure 6 directly targets the contribution of evolution. The ablation in Table 2 and the case studies in Appendix B provide useful qualitative evidence that skill fusion does not simply accumulate prompts. If the empirical claim is confirmed with proper uncertainty quantification and with exact prompts and skill records released, NeSy-Spatial would be a meaningful step toward agents that accumulate and reuse spatial procedures. However, the current evidence is a single run per condition, and the margins over strong baselines are small on GPT-5.4; the headline 'consistently improves' claim is not yet statistically supported. The manually specified, benchmark-specific seed skills and prompt templates also mean the results do not yet demonstrate transfer to unseen task families, a limitation the paper should acknowledge more directly.","major_comments":[{"comment":"The central empirical claim rests on single-run point estimates. The paper reports no standard deviations, no number of seeds, no significance tests, and no sampling temperature or number of trials. On GPT-5.4 the margins over the strongest baseline are only 1.50 points on MMSI and 0.49 points on MindCube, and NeSy-Spatial trails GCA by 0.73 points on OmniSpatial. The FTCS and Python success rates in Figure 5 and the Frozen comparison in Figure 6 are likewise single trajectory outcomes. Because the abstract and conclusion claim 'consistent' improvement, the authors should provide multiple seeds, bootstrap or permutation confidence intervals, or paired tests over the online stream, and report the sampling temperature and number of trials. This is load-bearing for the headline claim, not a cosmetic issue.","section":"§4.2, Table 1, Figures 5-6"},{"comment":"There is a reproducibility gap. Section 4.1 promises that 'We provide all prompts used in the paper, the complete set of evolved rules, and the remaining hyperparameter settings in the appendix,' but Appendix D ends with 'Full prompts will be available after the code release,' and no code or release link is given. The method is prompt- and LLM-driven; the abridged prompt templates and representative skill kernels in the appendix cannot be used to reconstruct the exact system. The full prompts and complete evolved skill records should be included in the appendix, or a working code release should be provided, before the empirical claims can be independently checked.","section":"§4.1 and Appendix D"},{"comment":"The abstract's statement that NeSy-Spatial 'consistently improves reasoning accuracy' is stronger than what the reported single run supports. Table 1 shows that on GPT-5.4 OmniSpatial, NeSy-Spatial trails GCA (60.27 vs. 61.00), and the MMSI and MindCube margins are 1.50 and 0.49 points. The conclusion's 'best aggregate accuracy in five of six settings' is an accurate description of the point estimates, but it should be the central claim rather than 'consistent improvement,' or it should be accompanied by the uncertainty analysis requested in the first comment.","section":"Abstract and §5"},{"comment":"The evaluation does not isolate the contribution of self-evolution from the contribution of dataset-specific engineering. The initial library is seeded with hand-written skills per benchmark (Nseed=3 for MMSI and MindCube, Nseed=4 for OmniSpatial), the prompt templates encode task-specific semantic rules such as 'one aggregated detection cannot fill multiple semantic roles,' and Table 4 sets dataset-specific maximum turns. Because the Frozen baseline and all ablations use the same seeds, they measure the effect of evolution given these priors, not the transferability of the framework to unseen task families. To support the claim that the agent accumulates reusable spatial procedures generally, the authors should add a cross-dataset or seed-variation experiment, for example seeding the library for one benchmark and evaluating on another, or varying the seed-skill set, and report sensitivity to these choices.","section":"§4.1, Appendix A.1, Appendix D"}],"minor_comments":[{"comment":"The sentence 'The same hyperparameters are used across all datasets and LVLM backbones' is contradicted by the dataset-specific values of Nseed (3/3/4) and by Table 4's dataset-specific maximum turns; please clarify which settings are shared and which are dataset-specific.","section":"§4.1"},{"comment":"In the provided version, the table cells run together and the bold/underline convention is hard to follow; please ensure the camera-ready table has clear column alignment and a footnote defining how 'Overall' is computed.","section":"Table 1"},{"comment":"Because each point in a cumulative accuracy curve includes all earlier samples, consecutive points are strongly correlated, so the observation that the curves become stable over time is expected; please note this non-independence and add confidence bands or a bootstrap estimate if possible.","section":"Figure 6"},{"comment":"The evolved skill inventory in Appendix C is useful, but the listed skills are only names; including one complete post-evolution skill record, with the full kernel, empirical statistics, and timestamps, would make the fusion and pruning claims easier to verify.","section":"Appendix C"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper applies skill self-evolution to tool-augmented spatial reasoning, with a two-type skill library (Tool-Use and Geometry) and an online buffer-then-evolve loop. That combination is genuinely new, and the design is careful. The problem is that the headline accuracy claim is supported by single-run point estimates with no error bars, and on GPT-5.4 the decisive margins are 1.50, 0.49, and -0.73 points. If run-to-run variance is anywhere near those numbers, the \"consistent improvement\" story does not hold.\n\nWhat is actually new: the typed atomic instructions with local verifiers, the split into Tool-Use and Geometry skills, and the pruning/retention rule are well thought out. The test-then-update protocol and the Frozen ablation are good controls: they show the label is not used to score the same prediction that produced it, and that online evolution does something beyond the seed skills. The case studies in Appendix B give concrete, readable evidence of how skills evolve, and the skill inventories show the library grows and prunes rather than accumulating indiscriminately. The paper is honest about its setup and shares core prompts, though full prompts are deferred to code release.\n\nWhere it is soft: the empirical core. No error bars, no multiple seeds, no significance tests, no stated sampling temperature. The GPT-5.4 margins are thin and one is negative; the Gemini gains are larger, but a single run cannot separate a real effect from API stochasticity or a lucky stream order. The Frozen comparison is encouraging but also single-run. This is a load-bearing gap because the central claim is empirical. Hyperparameters are hand-set without sensitivity analysis, and the seed skills are designed with knowledge of the benchmarks. That is not circular under test-then-update, but it means the method starts with some prior access to the task families.\n\nWho this is for: people working on tool-augmented reasoning, skill induction, or embodied spatial tasks. It deserves a serious referee: the idea is solid, the engineering is detailed, and the flaws are addressable with more runs and a released code/prompt artifact. I would not desk-reject it, but I would insist on error bars and a released prompt/code artifact before accepting. My verdict is conditional: the framework is promising, the evidence as reported is not yet decisive.","headline":"Solid engineering contribution with a careful test-then-update protocol and Frozen control, but the headline accuracy claim rests on single-run point estimates with thin or negative margins on GPT-5.4.","tokens_in":25253,"tokens_out":2039,"duration_ms":23001},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-evolving neuro-symbolic skill library, built from typed tool and geometry instructions, improves spatial reasoning accuracy and tool reliability across three benchmarks and two backbones.","keywords":["neuro-symbolic reasoning","spatial reasoning","tool augmentation","skill self-evolution","vision-language models","geometric computation","online learning","tool-use skills"],"falsifier":"Rerun the three benchmarks under several random orders of the online stream and multiple independent runs of NeSy-Spatial and its closest baselines. If the reported accuracy gaps shrink below the run-to-run variance, or if a baseline outscores NeSy-Spatial on more than one of the six backbone–dataset settings, the 'consistently improves' claim would be refuted.","tokens_in":24274,"feed_emoji":"🧭","tokens_out":8528,"duration_ms":77312,"temperature":0.7,"pith_summary":"NeSy-Spatial claims that fine-grained spatial reasoning can be improved by having a vision-language agent accumulate reusable 'skills' from its own tool-use history, instead of planning tool calls from scratch or following a fixed pipeline. The paper defines skills as typed atomic instructions—individual tool calls and geometric code blocks—composed into Tool-Use Skills for execution order and Geometry Skills for verifiable computation. During inference the agent retrieves and executes relevant skills in a closed loop; during evolution it analyzes buffered successes and failures, fuses supported fragments, and prunes unreliable or inactive entries. On MMSI, MindCube, and OmniSpatial with two backbones, it reports the best aggregate accuracy in five of six settings and the highest average first-pass tool-chain success and Python success rates. If the claim holds, spatial reasoning agents can improve with experience and transfer procedures across tasks rather than replanning from nothing each time.","feed_headline":"Self-evolving skills lift spatial reasoning in five of six settings","feed_subtitle":"The NeSy-Spatial agent composes tool-use and geometry skills from experience instead of planning from scratch.","key_machinery":"The load-bearing object is the skill library $\\mathrm{Lib} = (\\mathcal{A}, \\mathcal{S})$, where $\\mathcal{A}$ is a set of typed executable atomic instructions $a = (e_a, v_a)$ with local verifiers, and $\\mathcal{S} = \\mathcal{S}_{\\mathrm{tool}} \\cup \\mathcal{S}_{\\mathrm{geo}}$ contains high-level skills represented as node–edge–context triples $s = (V, E, \\kappa)$. Tool-Use Skills ($V \\subseteq \\mathcal{A}_{\\mathrm{tool}}$) encode which tools to call and in what dependency order; Geometry Skills ($V \\subseteq \\mathcal{A}_{\\mathrm{geo}}$) are reusable code kernels executed inside the Python tool, covering coordinate transforms, comparisons, and relation verification. The inference loop alternates skill retrieval, action selection, tool/code execution, and state update, so a loaded skill can be interrupted after any node. Evolution then performs trajectory analysis, extracting atomic-instruction updates and supported or invalid fragments; fusion edits the library by adding, deleting, or replacing subgraphs; and pruning removes entries using an error-ratio threshold, an inactivity limit, and a tenure-dependent retention factor.","core_discovery":"The central discovery is that a neuro-symbolic skill library can be made self-evolving for tool-augmented spatial reasoning. NeSy-Spatial represents skills as node-edge-context structures over typed atomic instructions, each with a local verifier: Tool-Use Skills organize controller-visible tool atoms into interruptible workflows with explicit dependencies, while Geometry Skills package reusable Python kernels for transformations, distance computations, and consistency checks. Under a test-then-update protocol, the agent evaluates each prediction before its label is used; after 20 completed trajectories it extracts atomic-instruction updates and supported or invalid fragments from the buffer, fuses them into the library, and prunes entries whose error rate or inactivity exceeds tenure-adaptive thresholds. The paper reports that this yields sustained cumulative accuracy gains over a frozen library—12.5 points on MindCube, 3.5 on MMSI, and 2.5 on OmniSpatial by the end of the stream—and that pruning keeps library growth bounded while retaining gains.","pith_inferences":["A natural extension the authors do not pursue is transferring the skill library across tasks: the atom-skill-prune machinery is domain-neutral, so the same loop might apply to tool-augmented table understanding or embodied navigation, where dependency-aware tool use matters.","The test-then-update protocol suggests the framework could operate in a fully online deployment without a training split, but the paper does not test robustness to different stream orders; that is a testable extension.","The reported gains in first-pass tool-chain success and Python success rate are larger and more consistent than the accuracy gains, so the most defensible near-term payoff may be reliability of tool execution rather than raw answer accuracy.","If future multi-seed evaluations support these point estimates, the practical recipe would be to seed a small library and let online experience refine it; if not, the framework still suggests that explicit dependency constraints are the key missing ingredient in from-scratch tool planners."],"forward_implications":["If the central claim is correct, spatial reasoning agents no longer need handcrafted pipelines per benchmark; a small seed library plus online evolution can reach or exceed fixed-pipeline accuracy on multiple datasets.","Tool execution becomes more reliable: higher average first-pass compositional tool-chain success and Python success rates mean fewer failed tool chains and fewer regenerated code executions, reducing cost and latency in deployed agents.","The same hyperparameters across datasets and backbones suggest the framework transfers without per-task tuning, a useful property for new spatial tasks.","Because evolution uses both successful and failed trajectories, the method can recover from recurring failures by adding dependency steps, as shown by the turn-then-move branch extension on MindCube.","The pruning mechanism prevents unbounded library growth, so accumulated experience remains bounded and retrieval context stays compact."],"supporting_citations":[{"why":"Provides the MMSI benchmark (multi-image spatial intelligence) used for evaluation, with Attribute, Motion, Positional Relation, and Multi-Step Reasoning subsets.","marker":"Yang et al. 2025b"},{"why":"Provides the MindCube benchmark with Rotation, Around, and Among subsets used to measure accuracy and skill evolution.","marker":"Chen et al. 2026b"},{"why":"Provides the OmniSpatial benchmark, evaluated on Dynamic Reasoning and Perspective Taking subsets.","marker":"Jia et al. 2025"},{"why":"Supplies the SpatialScore baselines SpaAge-PE and SpaAge-ReAct, which plan tool calls directly over a toolbox and are compared against NeSy-Spatial.","marker":"Wu et al. 2026"},{"why":"Supplies the GCA baseline, a geometrically-constrained agent that formalizes problems before invoking tools, and also the source of the OmniSpatial subset protocol.","marker":"Chen et al. 2026a"},{"why":"Supplies the LAST baseline and the notion of predefined spatial skills in a tool-augmented setting; its tool-chain success and Python success rates are directly compared.","marker":"Tian et al. 2026"},{"why":"Supplies the AWM baseline (Agent Workflow Memory), a general inductive agent whose workflow-memory approach is compared in the online setting.","marker":"Wang et al. 2024"},{"why":"Supplies the ASI baseline (Inducing Programmatic Skills for agentic tasks), representing experience-driven skill induction that NeSy-Spatial extends to spatial tool use.","marker":"Wang et al. 2025b"},{"why":"Supplies VGGT, the visual geometry grounded transformer used as the reconstruction backend behind the GeometricReconstructor tool atom.","marker":"Wang et al. 2025a"}],"fun_headline_variants":["Self-evolving skills boost spatial reasoning with tools","Agent evolves own tool skills for spatial tasks","Neuro-symbolic skills self-improve for spatial reasoning","Spatial reasoning gets self-evolving tool composition","Evolving skill library sharpens tool-augmented reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy gaps the paper reports are real and stable: the experiments are single-run point estimates without error bars, multiple seeds, or significance tests, and on one GPT-5.4 benchmark NeSy-Spatial's margin over the best baseline is less than two points while it trails on another.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving skills boost spatial reasoning with tools","Agent evolves own tool skills for spatial tasks","Neuro-symbolic skills self-improve for spatial reasoning","Spatial reasoning gets self-evolving tool composition","Evolving skill library sharpens tool-augmented reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1908,"prompt_tokens":942,"completion_tokens":966,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":893}},"tokens_in":558,"tokens_out":966,"duration_ms":12259,"temperature":1.0,"reasoning_tokens":893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:37:14.007793+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Rerun the three benchmarks under several random orders of the online stream and multiple independent runs of NeSy-Spatial and its closest baselines. If the reported accuracy gaps shrink below the run-to-run variance, or if a baseline outscores NeSy-Spatial on more than one of the six backbone–dataset settings, the 'consistently improves' claim would be refuted.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}