{"id":"509c8dff-83a4-4fa5-995b-6ec36b679013","arxiv_id":"2507.11079","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A VLM-plus-LLM commander trained with an expert rule system reaches 80-83% win rates in simulated multi-UGV confrontations, beating rule, RL, and vision-only baselines.","lead":"This paper builds a two-stage tactical commander for unmanned ground vehicles: a vision-language model turns the battlefield image into a text description, and a lightweight language model issues movement and combat orders from that description. In the authors' own simulator, the combined system wins 80-83% of engagements against rule-based, reinforcement-learning, and vision-only baselines, while deciding faster.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The expert system is both the label generator and the Rule baseline, with undisclosed thresholds and no sensitivity analysis; the 80% win rate therefore lacks an independent tactical reference and is not yet established.","rationale":"The reader's conditional verdict seems right, and the identified weakest assumption is the same one I would stress. I considered whether the model beating its own expert teacher (83 vs 67 against Rule enemy; 80 vs 71 against RL enemy in Table 3) already dissolves the circularity concern; it does not fully, because it only shows the student is better than its teacher, not that the teacher is a competent baseline. The comparison is still internal to a simulator and rule set designed by the authors. The more specific defects—undisclosed expert thresholds, unspecified RL baseline, absence of error bars, and the questionable individual-threat definition in Eq. (4)—compound this rather than creating a separate contradiction. The paper's title and abstract claim 'over 80% win rate compared with baseline models,' but with only 10 seeds and no variance, that point estimate is not statistically supported. These are empirical and reproducibility gaps, not internal inconsistencies, so the appropriate verdict remains CONDITIONAL; the reader's verdict is unchanged.","tokens_in":13373,"tokens_out":6727,"duration_ms":90558,"concrete_test":"Request or independently re-implement the expert system, then run a sensitivity sweep over the Table 1 thresholds and Eqs. (4)-(8) weights (±20% perturbations), retraining the commander on each perturbed label set and evaluating against the original Rule/RL baselines. If the 80% win rate drops below 55% or shifts by more than ~15 percentage points under plausible perturbations, the reported advantage is an artifact of a single unvalidated rule configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 reports the headline numbers (83% vs Rule, 80% vs RL), but the Rule baseline is the same expert system whose Eqs. (4)-(8) and Table 1 generated the training labels, and the RL baseline is not described. None of the expert-system thresholds (θ, δ, ε, ω, γ, ζ, σ) is given, no sensitivity sweep is run, and results are 10-seed point estimates without variance. This is load-bearing because the entire training signal is a supervised imitation of this rule set; if the rule set is a weak or idiosyncratic tactical reference, the learned commander can appear strong in the authors' simulator without representing a generally superior tactic. The ablation result that Ours beats its own teacher (83 vs 67, 80 vs 71) is real evidence against pure mimicry, but it does not validate the teacher's quality. An additional internal red flag is Eq. (4): T(e_j) is defined as the total Gaussian threat field I(x) evaluated at e_j's position, summed over all enemies, rather than an isolated threat score for e_j; this makes the expert system an informal heuristic whose behavior could change substantially under correction or re-tuning. Without external validation, the central 'over 80%' claim is conditional on a single, unvetted rule implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hierarchical commander for multi-UGV confrontation, combining a vision-language model (VLM) for scene understanding and a lightweight large language model (LLM) for tactical reasoning. The two modules communicate through a structured semantic space, and the system is trained using labels generated by a handcrafted expert rule system (Section 3.3). Simulation experiments with 5v5 confrontations report win rates of 83% against a rule-based enemy and 80% against a reinforcement-learning enemy, with additional ablations and a generalization study to 7v7 and 9v9 settings. The authors argue that separating perception from reasoning improves perceptual accuracy, reduces decision time, and yields better tactical performance than single-VLM or rule-based baselines.","tokens_in":13717,"tokens_out":3890,"duration_ms":43741,"significance":"If the reported results are robust, the paper demonstrates a plausible and interpretable architecture for tactical decision-making in adversarial multi-agent scenarios, with a clean separation of perception and reasoning via a shared semantic space. The use of a lightweight LLM as a strategic planner is timely and the modular design is appealing. However, the current evidence is not yet convincing: the central comparison is potentially circular because the expert rule system that generates training labels also serves as the Rule baseline; statistical uncertainty is not reported despite only 10 episodes per condition; the RL baseline is not described; and the generalization results are unreadable in the provided figure. These issues prevent the paper from establishing its headline 'over 80% win rate' claim as a general result. The paper does provide a concrete experimental pipeline and an honest discussion of perception limitations, which are strengths, but the validation needs substantial strengthening.","major_comments":[{"comment":"The expert system described in Section 3.3 (Eqs. 4–8 and Table 1) generates the training labels for the proposed commander and also serves as the 'Rule' baseline in Table 2. This creates a circular comparison: the learned model is trained to imitate these very rules, so a higher win rate against the Rule baseline chiefly indicates that the model has learned to deviate from its teacher, not that the tactics are strong in any absolute sense. The paper should provide an independent tactical reference (e.g., a standard baseline from the literature, a human-expert evaluation, or a much stronger rule set) and a sensitivity analysis of the expert-system parameters to demonstrate that the reported advantage is not an artifact of a weak or idiosyncratic rule implementation.","section":"Section 3.3, Table 1, and Section 4.2, Table 2"},{"comment":"The experimental setup states that each condition is repeated with ten independent experiments, but Table 2 reports only point estimates of win rate, survival rate, and decision time, with no variance, confidence intervals, or significance tests. With n=10 episodes per condition, the differences between, for example, 83% and 67% win rates (vs. Rule) or 80% and 71% (vs. RL) may be within sampling noise. The authors should report standard deviations or confidence intervals and perform a suitable statistical test (e.g., bootstrap or Mann–Whitney U) to support the claim of superiority.","section":"Section 4.1 and Table 2"},{"comment":"The reinforcement-learning baseline (denoted 'RL') is never described. The paper does not state which RL algorithm is used, the network architecture, the observation and action spaces, the reward function, or any training hyperparameters. Without this information, the comparison 'Ours vs. RL' is uninterpretable, and the reader cannot assess whether the RL baseline is a strong or weak representative of the method class. The authors must provide a complete description of the RL baseline, consistent with how the VLM baseline is described.","section":"Section 4.2, Table 2"},{"comment":"Perception metrics P, R, and RH (perception accuracy, recall rate, and hallucination rate) are mentioned but never formally defined. In particular, it is unclear what constitutes a 'hallucination' in this context, how ground truth is obtained for the perception evaluation, and over which units or frames the precision and recall are computed. Without precise definitions, these metrics cannot be reproduced or meaningfully compared across systems.","section":"Section 4.2, Table 2"},{"comment":"Figure 6, which reports the generalization results for 7v7 and 9v9 scenarios, is unreadable: the figure text is corrupted in the manuscript, and the underlying numerical values are not given in the text or a table. Since the robustness of the method at larger scales is one of the paper's claims, the authors must provide a clean, legible figure and report the exact numbers for perception accuracy, decision time, survival rate, and decision gain at each scale.","section":"Figure 6 and Section 4.3"},{"comment":"The threat score T(e_j) is defined as the total Gaussian threat field I evaluated at the position of e_j, where I sums contributions from all enemies el (including e_j itself). This makes T(e_j) a measure of the ambient threat at that location rather than the strategic impact attributable to enemy e_j. Because T(e_j) is subsequently used in the attack-cost rules (e.g., R8 in Table 1), the definition should be corrected to isolate e_j's own contribution, or the authors should explicitly explain why the aggregate field is the intended interpretation.","section":"Equation (4)"}],"minor_comments":[{"comment":"The abstract claims a 'win rate of over 80%', but Table 2 reports exactly 80% against the RL enemy; consider using 'at least 80%' or 'over 80% in one scenario and 80% in the other' to match the data.","section":"Abstract and Table 2"},{"comment":"Many thresholds and weights (θ, δ, ε, ω, γ, ζ, σ) are introduced without numerical values or a sensitivity analysis. Providing the chosen values (or a repository with configuration files) would greatly improve reproducibility.","section":"Section 3.3, Equations (4)–(8)"},{"comment":"The symbol σ is used both as the Gaussian bandwidth in Eq. (4) and as the logistic function in Eq. (5); the dual use is confusing and should be disambiguated.","section":"Equations (4) and (5)"},{"comment":"Equation (4) uses an aspect-ratio parameter ρ for the map, but the arena dimensions are given (30 m × 16 m) and ρ is not defined; please clarify its value or relation to the arena. ","section":"Section 4.1"},{"comment":"The fine-tuning description mentions LoRA adaptation, DPO alignment, and cosine learning rate schedules, but omits concrete hyperparameters (e.g., LoRA rank, learning rate, batch size, number of epochs). These details are necessary for reproducibility.","section":"Section 3.4"},{"comment":"The rows for the Rule-based allied model (vs. Rule and vs. RL) show dashes for all metrics; the authors should indicate that these are not applicable because the rule model has no perception module, or provide equivalent numbers where possible.","section":"Table 2"},{"comment":"The figures that illustrate a confrontation example and the survival curves are difficult to read in the current resolution; enlarging the panels or adding higher-resolution versions would help the reader verify the qualitative claims.","section":"Figures 4 and 5"},{"comment":"There are several minor grammatical and stylistic issues (e.g., 'the' vs. 'a', missing spaces) and some reference entries lack page ranges; a careful copyedit would improve readability.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper does something genuinely new: it builds a hierarchical perception (unit/local/region) with a VLM and a lightweight LLM planner, trained by LoRA and DPO against a hand-built expert system, and tests it in a multi-UGV simulator. The ablation shows the full system beats the single VLM and the expert-only baseline (83% vs 67% and 80% vs 71% win rates), which is real evidence that the LLM adds tactical value beyond imitation.\n\nSecond, the load-bearing comparison is circular in a way that matters. The expert system in Section 3.3 generates the training labels and is also the Rule baseline in Table 2. The model is trained to imitate this rule set, so outperforming it shows the LLM generalizes beyond its teacher, but it does not validate that the teacher is a good tactical reference. That would require external validation or at least a sensitivity sweep over the expert thresholds (θ, δ, ε, ω, γ, ζ, σ), none of which are disclosed. The threat score in Eq. (4) is also strange: T(e_j) is the sum of the Gaussian threat fields from all enemies evaluated at e_j, not an isolated threat score. That makes the expert heuristic informal and its behavior sensitive to re-tuning.\n\nOther soft spots: the RL baseline is never described; there are no confidence intervals or significance tests despite only 10 runs per cell; perception metrics P/R/RH are not defined; and Figure 6, which reports generalization to 7v7 and 9v9, is corrupted in the PDF and unreadable. The authors do acknowledge perception errors in dense scenarios, which is honest, but they don't quantify the impact on tactics.\n\nWhat the paper does well: the architecture is clean, the ablation is the right kind of experiment, and the decision time reduction (roughly 25%) is a plausible benefit of separating perception from reasoning. The generalization claim is plausible from the text even if the figure is broken.\n\nMy take: this is a legitimate extension of existing VLM+LLM planners (ReplanVLM, GameVLM, LLCoach) with a new application and a clear training recipe. The central 80% win-rate claim is not yet established because it rests on an unvetted expert system and a hidden baseline. But the issues are addressable: release the code and tuned thresholds, add error bars, describe the RL baseline, and fix Figure 6. A serious referee should see it, but it needs a major revision before I'd trust the numbers. If the authors do that, it could be a solid contribution to military simulation and game AI.\n\nRecommendation: send to peer review. It deserves referee time because the idea is sound and the empirical gap is fixable, not fatal.","headline":"A sensible VLM+LLM commander that beats its own rule-based teacher in a bespoke simulator, but the expert system doubles as the baseline and the thresholds are hidden, so the headline win rate is not yet established.","tokens_in":14250,"tokens_out":2169,"would_cite":false,"duration_ms":24124,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model commander that separates perception from planning beats rule-based and reinforcement-learning baselines in simulated UGV confrontations.","keywords":["multi-agent confrontation","vision-language model","large language model","tactical decision","unmanned ground vehicles","expert system","interpretable AI","reinforcement learning baseline"],"falsifier":"Run the same VLM-plus-LLM commander against an expert system whose thresholds and weights are tuned adversarially, or against a human expert, and measure whether the reported 80 percent-plus win rate survives; a drop to near 50 percent would indicate the advantage comes from the specific rule set rather than from the learned reasoning.","tokens_in":13137,"feed_emoji":"🤖","tokens_out":5880,"duration_ms":56085,"temperature":0.7,"pith_summary":"The paper tries to establish that a commander built from a vision-language model (VLM) for scene understanding and a lightweight large language model (LLM) for tactical planning can make interpretable, adaptive decisions in multi-UGV confrontations. It proposes a full-chain pipeline: the VLM converts a bird's-eye image into hierarchical semantic descriptions, and the LLM outputs per-agent instructions such as attack, support, or retreat. An expert system, active only during training, generates rule-based labels and preference pairs that align the two modules. In a 30 m by 16 m simulator with 5-on-5 UGVs, the approach reports win rates of 83 percent against a rule-based enemy and 80 percent against a reinforcement-learning enemy, with higher survival rates and lower decision times than a single-VLM baseline. If correct, the result supports the idea that separating perception from strategic reasoning, with shared semantic language, is a practical route to transparent tactical AI.","feed_headline":"VLM plus LLM commander wins 80% of simulated UGV battles","feed_subtitle":"Separating visual perception from tactical reasoning gives an interpretable commander that beats rule and RL baselines.","key_machinery":"The load-bearing mechanism is the expert system: a set of heuristic rules (Table 1) built on three computed metrics—threat score $T(e_j)$ from Eq. 4, danger value $D(a_i)$ from Eq. 6, and attack cost $C(a_i,e_j)$ from Eqs. 7–8—that convert structured perception into tactical labels. During training, these labels supervise the VLM's semantic outputs and provide chain-of-thought exemplars plus DPO preference pairs for the LLM, aligning the two modules within a shared semantic space $S = \\{S_u, S_l, S_r\\}$ at unit, local, and region levels. The same metric-driven rules also serve as the Rule baseline in the comparison, which is why the reported win rates should be read as a comparison against the authors' own rule set.","core_discovery":"The central claim is that a VLM-plus-LLM commander, trained with an expert system's rule-based supervision, outperforms both handcrafted rules and a single VLM in autonomous ground-vehicle confrontations. The paper reports, in Table 2, an 83 percent win rate against the Rule enemy and 80 percent against the RL enemy, a survival rate of 0.90 and 0.86 respectively, and an average decision time of about 15.4 seconds and 14.5 seconds, roughly 25 percent faster than the single-VLM baseline. The authors interpret this as evidence that offloading tactical reasoning to a language planner lets the visual module focus on perception, improving both accuracy and speed, and that the LLM can tolerate perception noise and produce strategic formations such as encirclement. They also report that ablations removing the LLM or the expert system degrade performance, and that the advantage persists when the scenario is scaled from 5-on-5 to 7-on-7 and 9-on-9.","pith_inferences":["The win criterion in the simulator is complete elimination of one team; real engagements often involve partial objectives, asymmetric capabilities, or retreat rules, so the 80 percent figure does not directly transfer to deployment settings.","Because the expert system is also the Rule baseline, the comparison measures the gap between a learned policy and its own teacher; a stronger independent baseline (for example, a different rule set or a human expert) would clarify how much genuine tactical novelty the LLM adds.","The semantic interface could be ported to other multi-agent adversarial domains where bird's-eye perception is available, but the expert system's equations and thresholds would need re-derivation for those dynamics, so the method is not immediately drop-in general.","A testable extension is replacing the expert system with a learned reward model or on-policy reinforcement learning fine-tuning, which would test whether rule-based supervision is necessary or merely convenient."],"forward_implications":["If the reported win rates hold, separating VLM perception from LLM planning is a viable architecture for adversarial multi-agent command, offering a balance between computational cost and decision quality.","The modular design means perception and planning can be upgraded independently: a better VLM or a stronger LLM should improve performance without changing the semantic interface.","The expert-system training paradigm offers a way to inject human tactical knowledge into LLM decision-making without hand-designing every rule, since the LLM can generalize beyond the rule set.","The observed tolerance to perception noise suggests that robust tactical reasoning can partially compensate for imperfect localization, which matters for real deployments where perception errors are inevitable.","Scaling results from 5-on-5 to 7-on-7 and 9-on-9 indicate the approach maintains its advantage as agent density grows, although decision time increases with scale."],"supporting_citations":[{"why":"Supplies the hierarchical rule-based decision framework that motivates the need for adaptable, non-rule commands.","marker":"[1]"},{"why":"Represents reinforcement-learning confrontation methods whose action-level focus this work contrasts with.","marker":"[2]"},{"why":"Gives the hierarchical reinforcement learning baseline whose strict interfaces motivate the flexible semantic interface.","marker":"[5]"},{"why":"Demonstrates ReplanVLM bridging planning and real-world environments, grounding the shared semantic space idea.","marker":"[7]"},{"why":"Shows multi-role LLM planning in robot soccer, the direct template for the role-based LLM commander.","marker":"[8]"},{"why":"Establishes that language models can reach human-level strategic reasoning in adversarial games, supporting the LLM as a strategic core.","marker":"[10]"},{"why":"Provides the VLM chain-of-thought decision-making method whose perception-to-action loop this work extends into a commander role.","marker":"[15]"},{"why":"Introduces expert agents evaluating plans, which this work adapts into an expert system that guides planning rather than evaluating.","marker":"[16]"}],"fun_headline_variants":["VLM+LLM commander wins 80% of simulated UGV battles","Vision-language commander beats rules and RL in UGV fights","LLM-led commander reaches 80% win rate in multi-UGV sims","Hybrid VLM-LLM commander achieves >80% win rate in UGV war games","Tactical AI commander uses VLM and LLM for faster, smarter UGV decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expert system's handcrafted rules encode genuinely good tactics, because those same rules generate the training labels and also serve as the Rule baseline that the method is compared against.","fun_headline_variants_meta":{"raw":{"variants":["VLM+LLM commander wins 80% of simulated UGV battles","Vision-language commander beats rules and RL in UGV fights","LLM-led commander reaches 80% win rate in multi-UGV sims","Hybrid VLM-LLM commander achieves >80% win rate in UGV war games","Tactical AI commander uses VLM and LLM for faster, smarter UGV decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000257,"raw_usage":{"total_tokens":1558,"prompt_tokens":907,"completion_tokens":651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":523,"tokens_out":651,"duration_ms":7508,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:16:41.387690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same VLM-plus-LLM commander against an expert system whose thresholds and weights are tuned adversarially, or against a human expert, and measure whether the reported 80 percent-plus win rate survives; a drop to near 50 percent would indicate the advantage comes from the specific rule set rather than from the learned reasoning.","supporting_citations":[{"cited_title":"IEEE Robot","cited_arxiv_id":null,"evidence_quote":"Demonstrates ReplanVLM bridging planning and real-world environments, grounding the shared semantic space idea."},{"cited_title":"Robot World Cup15570, 176–188 (2025) https://doi.org/10.1007/ 978-3-031-85859-8 15","cited_arxiv_id":null,"evidence_quote":"Shows multi-role LLM planning in robot soccer, the direct template for the role-based LLM commander."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the VLM chain-of-thought decision-making method whose perception-to-action loop this work extends into a commander role."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces expert agents evaluating plans, which this work adapts into an expert system that guides planning rather than evaluating."}],"review_version":1}