{"id":"5e92e68f-c978-4a1a-b067-be6b423b878a","arxiv_id":"2507.10778","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM agent equipped with lightweight distance and inclusion perception models achieved 95.86% accuracy on the 2025 AI City Challenge warehouse spatial QA benchmark, ranking first.","lead":"This paper describes a chatbot-like system that answers spatial questions about warehouse scenes by combining a large language model with small computer-vision tools for measuring distances and checking whether objects are inside regions. It took first place in the 2025 AI City Challenge Track 3 with 95.86% accuracy on the official warehouse benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No isolated accuracy is reported for the distance and inclusion perception models, so the 95.86% leaderboard result cannot be decomposed into perception quality versus LLM agent reasoning quality.","rationale":"The paper's strongest claim, a 95.86% leaderboard accuracy on an official benchmark, is supported by the external competition result. The reader's conditional verdict is appropriate because the paper provides no isolated accuracy for the learned perception modules and no ablations that would separate perception errors from LLM reasoning quality. My stress-test concern aligns with the reader's weakest assumption: the two ResNet-50 models trained only on the training split must transfer to the hidden test, and their error rates are unreported. This is a documentation gap rather than a demonstrated flaw; the leaderboard number itself is credible. A concrete validation-set analysis with per-module and per-question-type results, plus an oracle ablation, would settle whether the agent architecture or the perception modules are responsible for the performance. Since the reader already recommends CONDITIONAL, my independent read does not move the verdict. No ad hominem is intended; the critique targets the evidence provided, not the authors' integrity.","tokens_in":6298,"tokens_out":3198,"duration_ms":40432,"concrete_test":"On the public validation split (1.9K QA pairs), report: (1) distance model Acc@10 and MAE; (2) inclusion classification accuracy; (3) end-to-end agent accuracy per question type; and (4) an ablation that replaces each perception API with ground-truth oracle values. If the oracle ablation improves accuracy by less than about one point, perception errors are negligible; if distance accuracy is low but end-to-end accuracy is high, the benchmark's weighting hides perception failures. Also report a code commit hash to make the pipeline reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 1st-place 95.86% hidden test accuracy. The leaderboard result is externally verifiable, so the factual claim is credible. However, the paper's interpretation — that an LLM agent with lightweight perception APIs outperforms MLLM finetuning — depends on the two trained ResNet-50 modules (distance regression in Sec. 3.2 and inclusion classification in Sec. 3.3) transferring to the hidden test distribution and contributing correctly. The paper reports neither validation accuracy nor per-module accuracy for these models. Since the end-to-end accuracy is a composition of mask parsing, LLM function-calling and reasoning, and perception predictions, a 95.86% system number cannot reveal whether the agent would remain strong if the distance model's MAE degrades on unseen warehouse layouts. The absence of ablations (e.g., replacing the distance API with an oracle) means the claim that the agent's reasoning, rather than the perception modules, drives performance is unsupported. The paper also omits per-question-type breakdowns; because the benchmark metric is a weighted average over four question types, the headline could be dominated by easy counting queries. These are missing-evidence concerns, not contradictions, and they do not invalidate the leaderboard result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SpatialAgent, an LLM-agent system for warehouse spatial question answering. The system takes an RGB image, binary object masks, and a spatial question; a rule-based parser converts mask tags into object identifiers; a Gemini 2.5 Flash agent then performs multi-turn reasoning and calls API tools for distance estimation, inclusion classification, relative left/right positioning, and region queries. Distance and inclusion are computed by trained ResNet-50 models (with a second small-distance cascade model), while simpler spatial relations use mask-centroid heuristics. The authors report 95.86% weighted accuracy on the hidden test set of the 2025 AI City Challenge Physical AI Spatial Intelligence Warehouse benchmark, ranking 1st on the official leaderboard, and they argue this demonstrates a data-efficient alternative to large-scale MLLM finetuning. The code is publicly released.","tokens_in":6558,"tokens_out":4759,"duration_ms":59211,"significance":"If the reported numbers are accurate, the result is significant: it achieves state-of-the-art accuracy on an externally scored challenge leaderboard using a modular agent with lightweight perception models rather than finetuning a large multimodal model. The official leaderboard provides a credible, independently verifiable evaluation, and the open-source code and clear system description support reproducibility. The paper is also honest in its empirical framing: the perception models are trained on the benchmark training split and evaluated on the hidden test set, with no indication of circularity or test-set leakage. However, the central interpretation—that the LLM agent's spatial reasoning, rather than the trained perception modules, drives the high accuracy—is not yet supported by the presented evidence, and the absence of per-question-type and per-module breakdowns limits how much can be concluded from the aggregate score.","major_comments":[{"comment":"The paper reports only a single weighted-average accuracy over the four benchmark question types (spatial relations, multi-choice, distance estimation, counting), as described in Sections 4.1 and 4.2. Without a per-question-type breakdown, the 95.86% aggregate could be dominated by easier query types, so the claim that the agent demonstrates 'complex spatial reasoning' is not substantiated. I request a table reporting accuracy separately for each question type, and ideally a comparison with the runner-up on the same breakdown.","section":"Section 4.4, Table 1"},{"comment":"No isolated validation accuracy is reported for the two trained perception models: the distance regression networks F and F_small, and the inclusion classifier G. The end-to-end 95.86% accuracy is a composition of mask parsing, LLM function calling, and these perception predictions, so the aggregate number cannot reveal whether the agent's reasoning or the perception modules contribute most to performance. To support the paper's interpretive claims, please report the validation MAE/accuracy of F, F_small, and G, and include an ablation that replaces the perception APIs with oracle/ground-truth values to isolate the LLM agent's reasoning contribution.","section":"Sections 3.2 and 3.3"},{"comment":"The cascade threshold is described as 'whenever F predicts a value smaller than 3m, we use F_small to predict again', with the motivation that F is inaccurate for distances under 3m. However, no validation analysis of this threshold is given, and it is a free parameter of the system. A threshold sweep or per-distance-bin accuracy comparison between F and F_small would be needed to support the claimed improvement from the cascade.","section":"Section 3.2"},{"comment":"The stochastic nature of the LLM agent is not addressed: the temperature is set to 0.2 and a retry mechanism adds 128 tokens, but the paper reports a single leaderboard result without variance. Since the same question can yield different outputs across runs, a repeated evaluation on the validation set with mean and standard deviation (or at least a statement about run-to-run stability) would strengthen the claim that 95.86% is a reproducible system-level accuracy.","section":"Section 4.3"}],"minor_comments":[{"comment":"The typesetting of the L2 distance loss is corrupted in the manuscript; the expression should be written cleanly as ||D_hat - D_gt||_2^2.","section":"Equation (1)"},{"comment":"The focal loss hyperparameters alpha_t and gamma are named but their values are not reported; please provide the values used for training the inclusion classifier.","section":"Section 3.3"},{"comment":"The table caption should state the evaluation metric (weighted average success rate with Acc@10 for distance/counting questions) and the date on which the leaderboard was accessed, since leaderboards can change.","section":"Table 1"},{"comment":"The phrase 'data-efficient' is used to contrast with MLLM finetuning, but no quantitative comparison of training data size, parameters, or compute is given; adding such numbers would make the claim more precise.","section":"Abstract and Section 2.2"},{"comment":"The geometric heuristics for 'closest', 'most left', and 'middle' are described only in words; a precise definition (e.g., whether 'closest' uses predicted distances or image-plane centroid distances) would improve reproducibility.","section":"Section 3.1"},{"comment":"Reference titles have inconsistent capitalization (e.g., [2] 'Spatialvlm', [13] 'Vlm-grounder'); please normalize them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a well-scoped challenge-solution paper with a strong, externally verifiable empirical result and a clean system description. The main reason for major revision is not the leaderboard number itself, but the missing decomposition evidence: without per-question-type and per-module validation, the paper's interpretive claims about the agent's reasoning ability and data efficiency outrun what the data shows. The requested additions are feasible and would turn this into a solid contribution. No concerns about circularity or data leakage were found."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know one thing before anything else: the 95.86% leaderboard number is real. It is on the official 2025 AI City Challenge Track 3 board, and that is a strong form of evidence. The paper is not a clean scientific derivation, but it does not have to be. It is a systems report, and on those terms it does most things right.\n\nWhat is actually new here is the specific integration. None of the ingredients are new: LLM agents with function calling, ResNet-50 regression and classification, centroid-based left/right heuristics. But putting them together in a tool-calling loop and getting first place on this warehouse QA benchmark is a concrete achievement, and the paper explains the assembly clearly. The distance cascade (F and F_small) and the inclusion classifier are sensible engineering choices, and the authors release code. Credit is due for shipping a working system with an official result.\n\nNow the soft spots, in proportion. The biggest one is that the paper reports no isolated accuracy or validation numbers for the two ResNet-50 perception modules. The headline system accuracy is a composition of mask parsing, LLM reasoning, and perception outputs, and without per-module numbers or an oracle ablation you cannot tell how much each contributes. The stress-test note is right: the claim that the agent's reasoning, rather than the perception models, drives performance is unsupported. This is a missing-evidence concern, not a contradiction. The leaderboard number itself is credible.\n\nThe other soft spots are minor but real. There is no per-question-type accuracy breakdown, so the weighted average could be dominated by easy counting queries. There are no error bars or repeated runs, which matters because LLM temperature is 0.2 and there is a retry mechanism. The comparison to MLLM finetuning is indirect — the paper does not run a SpatialVLM-style baseline on the same test set, so the claim of outperforming large-scale MLLM finetuning is only inferred from the challenge context. And the benchmark is synthetic, so the conclusion's mention of generalizability is a stretch.\n\nWho is this for? People building LLM agents for spatial tasks, and anyone who wants a clear description of a winning challenge system. It deserves a serious referee — the result is externally verified and the method is described well enough to replicate — but a referee should ask for the missing ablations and per-type results. I would send it to peer review, likely with a request for revision rather than acceptance on first submission.\n\nNet: solid, honest, somewhat thin on analysis. Worth reading, worth citing for the system result, and worth engaging with in review.","headline":"This first-place challenge paper is a credible systems report whose headline number is externally verified, but the missing per-module ablations keep the result from being decomposable into perception versus reasoning quality.","tokens_in":7115,"tokens_out":1627,"would_cite":true,"duration_ms":21569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM agent with small perception tools beats MLLM finetuning on warehouse spatial QA, hitting 95.86 percent.","keywords":["LLM agent","spatial reasoning","warehouse spatial question answering","function calling","distance estimation","inclusion classification","AI City Challenge","multimodal large language models"],"falsifier":"Take the public validation split, isolate questions whose correct answer depends on the distance regressor or the inclusion classifier, and measure those categories' accuracy separately from questions answerable by the left/right centroid rule; if the perception-dependent categories are near chance while the overall score stays high, the 95.86 percent result would not demonstrate general spatial reasoning.","tokens_in":6117,"feed_emoji":"🤖","tokens_out":9595,"duration_ms":94900,"temperature":0.7,"pith_summary":"The paper tries to establish that complex spatial question answering in warehouses does not require large-scale finetuning of a multimodal large language model (an image-and-text model). It proposes SpatialAgent, a system in which a general-purpose LLM with function-calling abilities answers questions by invoking small, specialized perception tools: a distance regression model, an inclusion classifier, and simple geometric rules for left/right relations. On the 2025 AI City Challenge warehouse benchmark, the system reaches 95.86 percent accuracy on the hidden test set and ranks first, ahead of teams using the finetuning-heavy approach. The practical point is that spatial reasoning can be decomposed into cheap perception modules plus language-level reasoning, which avoids the massive training data and compute that prior spatial MLLMs needed.","feed_headline":"LLM agent with small tools tops warehouse spatial QA at 95.86%","feed_subtitle":"A general-purpose language model plus lightweight APIs outperforms huge multimodal finetuning on spatial questions.","key_machinery":"The load-bearing object is the SpatialAgent system itself: a function-calling LLM that maintains a structured message history and executes tool calls enclosed in execution tags before producing a final answer. Around it sit two lightweight perception modules, a cascaded ResNet-50 distance regressor trained with L2 loss (with a second model specializing in distances under three meters) and a ResNet-50 inclusion classifier trained with focal loss, plus rule-based geometric tools that use mask centroids for left/right and region queries. What carries the argument is the division of labor: the LLM handles question decomposition and spatial reasoning, while the small modules handle the measurements that reasoning depends on.","core_discovery":"The central discovery, on the paper's own terms, is that an LLM agent with tool interaction is enough to solve the warehouse spatial QA task at the top level. The system takes an RGB image, object masks, and a natural-language question; a rule-based parser rewrites masks into object identifiers; the LLM then plans multi-step reasoning and calls APIs such as distance, left/right, and inclusion checks. The distance models are ResNet-50 regressors, one general and one specializing in distances under three meters, and inclusion is a ResNet-50 binary classifier trained with focal loss. With this composition the agent obtains 95.8638 percent on the hidden test set, the top score in the challenge, which the paper presents as evidence that a data-efficient LLM agent can outperform or match large-scale MLLM finetuning on this benchmark.","pith_inferences":["A testable extension the paper leaves implicit is to swap the reasoning LLM for a cheaper or open-weight model; if accuracy barely moves, the method's strength lies mostly in the perception tools, not the particular LLM.","The paper reports no standalone accuracy for the distance and inclusion models, so the natural next measurement is to evaluate those modules alone and separate perception error from reasoning error in the 95.86 percent.","Because the benchmark is synthetic and supplies clean masks, transferring this system to real warehouses would require adding an object detector to produce masks; that is a follow-up the paper does not claim to solve."],"forward_implications":["The same recipe can be applied to other spatial QA benchmarks where masks or detections are available; one keeps the LLM general and retrains only the small perception APIs.","Accuracy on distance and counting questions becomes a direct function of the small regressors and classifiers, so improving those modules is the most direct route to pushing beyond 95.86 percent.","Adapting to a new warehouse layout or camera setup becomes cheap, because only the lightweight perception models need retraining rather than the entire multimodal model.","The multi-turn execution leaves an audit trail: every final answer is backed by the sequence of tool calls and intermediate results, which can be inspected when an answer is wrong."],"supporting_citations":[{"why":"Supplies the benchmark dataset and the official evaluation protocol on which the 95.86 percent result is measured.","marker":"[11]"},{"why":"Supplies the LLM used as the reasoning and function-calling agent.","marker":"[4]"},{"why":"Supplies the ResNet-50 backbone for the distance regression and inclusion classification modules.","marker":"[5]"},{"why":"Supplies the focal loss used to train the inclusion classification model.","marker":"[10]"},{"why":"Represents the large-scale MLLM finetuning paradigm the paper contrasts with its agent approach.","marker":"[2]"},{"why":"Represents the large-scale MLLM finetuning paradigm the paper contrasts with its agent approach.","marker":"[3]"}],"fun_headline_variants":["LLM agent with tools tops warehouse spatial QA","Data-efficient LLM agent beats MLLM finetuning on spatial QA","Tool-using LLM agent wins warehouse spatial challenge","Lightweight agent outscores huge multimodal models on spatial tasks","Small tools, top score: LLM agent on warehouse QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the two small distance and inclusion models, trained only on the benchmark's training split, stay accurate enough on the hidden test scenes that their errors do not dominate the 95.86 percent score.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent with tools tops warehouse spatial QA","Data-efficient LLM agent beats MLLM finetuning on spatial QA","Tool-using LLM agent wins warehouse spatial challenge","Lightweight agent outscores huge multimodal models on spatial tasks","Small tools, top score: LLM agent on warehouse QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001029,"raw_usage":{"total_tokens":4289,"prompt_tokens":855,"completion_tokens":3434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":3351}},"tokens_in":471,"tokens_out":3434,"duration_ms":26828,"temperature":1.0,"reasoning_tokens":3351,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:25:14.449218+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the public validation split, isolate questions whose correct answer depends on the distance regressor or the inclusion classifier, and measure those categories' accuracy separately from questions answerable by the left/right centroid rule; if the perception-dependent categories are near chance while the overall score stays high, the 95.86 percent result would not demonstrate general spatial reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset and the official evaluation protocol on which the 95.86 percent result is measured."},{"cited_title":"Spatial- rgpt: Grounded spatial reasoning in vision-language mod- els","cited_arxiv_id":null,"evidence_quote":"Represents the large-scale MLLM finetuning paradigm the paper contrasts with its agent approach."}],"review_version":1}