{"id":"2d7af22f-a34c-43ba-9f2f-b6ca3a1496ae","arxiv_id":"2608.03735","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Multilingual agent planning failures concentrate in five grounding categories, and making those categories explicit in a structured representation improves multi-agent task accuracy by up to 10 points.","lead":"This paper identifies five recurring ways that multilingual requests get mangled when an LLM planner converts them into executable agent plans, and introduces TART, a structured task representation that makes those five commitments explicit to the whole agent team. Across eleven languages, three model families, and two agent benchmarks, TART consistently improves accuracy, including a 5.6 point average gain on a state-of-the-art GAIA system.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing generic structured-representation control leaves TART's gains unattributed: the converter pass, not the taxonomy fields, may drive the +5.6-point improvement.","rationale":"The reader's weakest assumption correctly identifies the missing generic structured-representation control. The paper's empirical core is solid: TART improves accuracy consistently across three model families, two datasets, and multiple languages, with a second GPT-5-mini run in Appendix E.1 showing stability. Those results support a claim that 'adding a TART-style structured summary improves performance.' But the stronger, more interesting claim—that the taxonomy's specific five grounding categories are what make the representation useful—requires a control that separates the effect of 'taxonomy-aligned content' from the effect of 'any structured intermediate representation.' The ablation in Figure 5(b) cannot provide this: every variant includes the converter pass and a taxonomy-derived schema. The manuscript's own limitations section acknowledges related gaps (human judge validation on one model only; repeated runs only for GPT-5-mini), which further caution against over-attributing the mechanism. Therefore the verdict should remain conditional: accept the empirical gains provisionally, but require the generic-json control before treating the taxonomy as the demonstrated cause of the improvement.","tokens_in":21633,"tokens_out":2793,"duration_ms":38049,"concrete_test":"Add a generic-json control condition on GAIA-MAPS with GPT-5-mini: keep the same converter model, the same planner prompt injection, and the same downstream agents, but replace the five taxonomy fields with a fixed, non-taxonomy JSON schema of comparable length (e.g., 'summary', 'important details', 'constraints', 'steps', 'output') whose content is generated by the same converter model. Run all 165 tasks in each of the 11 languages under Baseline, TART, and Generic-JSON, and compare Exact Match. If Generic-JSON's mean gain over Baseline is within noise of TART's +5.6 points, the taxonomy's causal role is not supported; if TART significantly outperforms Generic-JSON, the attribution is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the five-category taxonomy is actionable, not merely descriptive. To support this, Section 5.1 and Figure 5(b) ablate which taxonomy fields are included in the TART representation, but every TART arm still performs the converter pass and produces a taxonomy-shaped JSON. There is no control condition in which the same model produces an equally structured but non-taxonomy representation before planning. The statement in Section 4 that 'using the same model as converter isolates the effect of structure' addresses model identity, not representational content: it does not rule out the possibility that any extra structured-summarization step improves planning by forcing the model to process the query more carefully. Thus the causal link between the specific taxonomy and the reported accuracy gains is untested. If a generic JSON summary yields the same improvement, the 'actionable diagnosis' claim reduces to a structured-prompting effect, and the taxonomy's five categories are not doing the causal work. This is the load-bearing gap between the diagnosis in Section 3.1 and the mitigation result in Section 5.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the request-to-plan boundary in multilingual multi-agent systems. It defines planning-grounding failures, derives a five-category taxonomy (entity, source, temporal, operation, answer-format) from 80 manually analyzed failures in six low-resource languages using a development set disjoint from the main evaluation, and uses an LLM-as-a-judge to show that the share of taxonomy-covered failures increases as Common Crawl resource availability decreases. It then proposes TART, a structured JSON representation whose fields correspond one-to-one to the taxonomy, injected into the planner and downstream worker prompts. On GAIA-MAPS, TART improves mean exact-match accuracy over baseline by 5.6 points for GPT-5-mini, 5.9 points for Mistral-Large-3, and 3.6 points for Qwen3-VL-235B-A22B; on MULTITAT, by 10.0 points (Mistral-Large-3) and 3.1 points (Qwen3-VL-235B-A22B). Ablations in Fig. 5(b) show cumulative gains from input, operation, and full fields.","tokens_in":21851,"tokens_out":5324,"duration_ms":61308,"significance":"If the causal role of the taxonomy is established, this is a valuable contribution: it connects a failure taxonomy to a concrete mitigation protocol, evaluates on two external benchmarks and three model families, reports repeated-run evidence for GPT-5-mini (20 of 22 language-run comparisons), provides an anonymous implementation, and includes a human-validated LLM judge. The empirical pattern is broad and consistent, and the authors are careful to keep the taxonomy derivation set disjoint from the evaluation set. The principal weakness is attribution: the current experiments do not separate the effect of the taxonomy's content from the effect of adding any structured representation pass.","major_comments":[{"comment":"The causal attribution of TART's gains to the five taxonomy categories is untested. Every TART variant in Fig. 5(b) includes a converter pass that produces a taxonomy-shaped JSON before planning; Baseline has no such pass. The statement in §3.3 that 'using the same model as converter isolates the effect of structure' controls for model identity, not representation content. A control condition with a generic structured summary (same converter pass, same JSON-like format, but fields not derived from the taxonomy) is needed to rule out the possibility that any structured restatement improves planning by forcing more careful reading or by adding inference compute. Without this control, the reported gains could be a structured-prompting effect, and the 'actionable diagnosis' claim is not established.","section":"§3.3; §5.1, Fig. 5(b)"},{"comment":"The resource-gradient claim — that taxonomy-covered failures increase as language resources decline — is produced by an LLM judge whose prompt contains the taxonomy's categories, definitions, and priority rules (App. G.1). Human validation is reported only for Mistral-Large-3 (macro-F1 0.906, App. G.4), yet the gradient is claimed for GPT-5-mini and Qwen3-VL as well. Because the judge's labels are the only evidence for the cross-model trend, validation on at least a stratified sample for the other two models, or a per-language breakdown of the human agreement, is needed before the claim is treated as established. The authors' limitation statement acknowledges calibration risk; the current experiments do not bound it.","section":"§5.2, Fig. 6, App. G.4"}],"minor_comments":[{"comment":"Human-annotation counts are inconsistent: §5.2 says 6 annotators verified 122 samples, while App. G.4 says 7 annotators produced 280 annotated samples and then reports 117 decisive verifications from 5 annotators. Please reconcile these numbers and report the final protocol clearly.","section":"App. G.4 vs §5.2"},{"comment":"Typos and rendering artifacts: 'Kygryz' in §5.2, 'drawning tasks' in Related Work, and 'Y oruba' with irregular spacing in several places. These should be corrected.","section":"Throughout"},{"comment":"Repeated-run stability is reported only for GPT-5-mini, and no confidence intervals or significance tests are given for the single-run results. At least a bootstrap or per-language variance estimate would help calibrate the strength of the claims for Mistral-Large-3 and Qwen3-VL.","section":"§5, Table 5"},{"comment":"The Level-3 plateau explanation uses reference-solution step/tool counts from GAIA annotations, which are suggestive but not directly measured in the agent's execution. Consider labeling this explanation as a hypothesis and testing it, e.g., by measuring success of TART on Level-3 tasks with shorter reference chains.","section":"§5, Fig. 5(c), Table 7"}],"recommendation":"major_revision","confidential_remarks":"This is a solid and well-executed paper whose main gap is fixable: the missing generic structured-representation control is a straightforward additional condition, and the human-annotation inconsistencies are presentation issues. I would not accept before the control is added, because the 'actionable' claim is the paper's central selling point. If the authors can show that a non-taxonomy structured summary does not reproduce the gains, the contribution is strong and could be accepted after moderate revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this paper has a clean, useful idea—five grounding-failure categories for the request-to-plan boundary in multilingual multi-agent systems—and it gets consistent accuracy gains from acting on them. Second thing: the main causal claim is not yet proven, because the TART condition always includes an extra structured-representation pass and there is no control with a similarly structured but non-taxonomy summary. That is the one load-bearing gap, and it is fixable.\n\nWhat is actually new: prior work documents multilingual degradation and agent failure taxonomies, but nobody has systematically categorized planner-specific grounding failures across languages and then built a mitigation protocol directly from the taxonomy. The five categories (entity, source, temporal, operation, answer format) overlap generic QA failure modes, but the planner-focused framing and the TART protocol are a real step. The empirical work is solid: three model families, two datasets, eleven languages, two independent runs for GPT-5-mini, and a human-validated LLM judge (macro-F1 0.906 on one model). Improvements are consistent: 20 of 22 language-run comparisons, no degradation except one small negative. The paper is honest about limitations—machine translation, single runs for most models, judge validation on one model.\n\nSoft spots, in proportion. The missing generic-json control is the big one. The statement that 'using the same model as converter isolates the effect of structure' addresses model identity, not representational content. A generic structured summary could plausibly force more careful processing and yield similar gains. The ablation in Fig. 5b varies taxonomy fields but every arm has the converter pass, so it cannot distinguish taxonomy-specific effects from a general structure effect. This should be a required addition. Minor issues: the annotator counts are inconsistent between the main text (6 annotators, 122 samples) and appendix G.4 (7 annotators, 280 samples, 117 decisive), and the taxonomy was derived only from low-resource failures, so its applicability to high-resource failures is less established. Neither is fatal.\n\nWho it's for: anyone building multilingual agent pipelines or studying cross-lingual LLM failures. It deserves a serious referee. A conditional accept with a required generic-json control condition would be the right outcome.","headline":"Solid empirical package with a real attribution gap: the TART gains are consistent, but without a generic structured-summary control the taxonomy's causal role is untested.","tokens_in":22352,"tokens_out":2189,"would_cite":true,"duration_ms":23462,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Five grounding categories diagnose and mitigate multilingual multi-agent planning failures.","keywords":["multilingual NLP","multi-agent systems","planning-grounding failures","failure taxonomy","low-resource languages","structured task representation","GAIA-MAPS","MULTITAT"],"falsifier":"Run the GAIA-MAPS evaluation with a control condition in which the converter produces a JSON summary with generic fields instead of the five taxonomy fields; if that control matches TART's gains, the taxonomy's specific content is not the active mechanism. The paper's ablation varies which taxonomy fields are included but never removes the converter, so this control would settle whether the diagnosis is actionable or the structure alone helps.","tokens_in":21546,"feed_emoji":"🌐","tokens_out":8000,"duration_ms":81972,"temperature":0.7,"pith_summary":"The paper argues that the dominant source of multilingual failure in LLM-based multi-agent systems sits at the boundary where a user request becomes a plan, and that these failures are not random: they fall into five recurring categories (entity, source, temporal, operation, and answer-format grounding). As the language used is less represented in training data, these five categories account for a growing share of all failed executions, especially operation and entity grounding. The authors then show that making these five commitments explicit in a structured task summary called TART, injected into planner and worker agents, improves accuracy across three model families, two benchmarks, and multiple resource levels, including a 5.6-point average gain on multilingual GAIA. If true, the result turns 'multilingual degradation' from a vague diagnosis into a specific, inspectable failure mode that a multi-agent architecture can mitigate without retraining.","feed_headline":"Plan-grounding taxonomy lifts multilingual agent accuracy 5.6 points","feed_subtitle":"Multilingual failures at the request-to-plan boundary are systematic enough to fix.","key_machinery":"The load-bearing object is TART, a Taxonomy-Guided Actionable Task Representation: an LLM-produced JSON with fields entities, time_constraint, source_constraint, attachment_type, operations, and answer_type, placed in one-to-one correspondence with the failure taxonomy. The mechanism is that this representation is injected into the system prompts of the planner, coordinator, and worker agents, so the semantic commitments remain visible through execution. The taxonomy itself is the other half of the machinery: it converts an observed failure into one of five named categories, and each category indicates which commitment the re-plan must restore.","core_discovery":"The central claim is that multilingual failures in LLM-based multi-agent systems are concentrated at the request-to-plan boundary and are systematic rather than diffuse: they recur as five planning-grounding failures. The paper defines these as mismatches between the user request and the generated plan such that even correct execution would not satisfy the request. The five categories are entity grounding (wrong, substituted, or hallucinated entities), source grounding (wrong source or wrong part of the source), temporal grounding (changed or dropped time constraints), operation grounding (wrong reasoning chain or objective), and answer-format grounding (correct content in the wrong output s","pith_inferences":["A natural test the paper does not run: compare TART against a generic structured JSON summary with non-taxonomy fields, to isolate whether the taxonomy's specific content or the mere structured restatement causes the gain.","The same semantic-contract mechanism could be applied beyond the request-to-plan boundary—for example, at worker handoff or answer composition—where constraints may degrade again after planning.","If the taxonomy is truly actionable, it suggests a targeted repair loop: classify a failed execution by grounding category and re-plan only the violated field, which would be cheaper than full replanning.","Machine-translated low-resource queries may understate or distort real failure patterns; naturally authored data in those languages would be a stronger test."],"forward_implications":["Multilingual agent systems can be improved by inserting an explicit semantic contract after the user request and before planning, without retraining or replacing the framework.","The taxonomy gives a reusable diagnosis: failures previously lumped as 'other' can be traced to one of five named grounding categories, and the category points to the fix—re-ground the entity, source, time, operation, or output format.","The benefit transfers across LLM backbones, datasets, and agent configurations, so the mechanism is not tied to one model or benchmark.","The ablation indicates that the operation field is the highest-value addition for low-resource languages, since operation grounding dominates their failures.","Gains concentrate at difficulty Levels 1 and 2; Level 3 remains flat, which the paper attributes to task complexity rather than an absence of correctable planning failures."],"supporting_citations":[{"why":"Supplies GAIA, the real-world agentic task benchmark whose multilingual version is the main testbed.","marker":"(Mialon et al., 2024)"},{"why":"Contributes GAIA-MAPS, the 11-language multilingual GAIA benchmark used for the primary accuracy comparisons.","marker":"(Hofman et al., 2026)"},{"why":"Provides OWL, the multi-agent workforce used as the baseline system that TART augments.","marker":"(Hu et al., 2025)"},{"why":"Contributes MULTITAT, the multilingual table-and-text QA benchmark used as the second dataset.","marker":"(Zhang et al., 2025b)"},{"why":"Provides per-language web-crawl shares used to order languages by resource availability.","marker":"(Common Crawl Foundation, 2026)"},{"why":"Establishes cross-lingual degradation in structured function calls, motivating the study of the request-to-plan boundary.","marker":"(Kulkarni et al., 2025)"}],"fun_headline_variants":["Planning-failure taxonomy lifts multilingual agent accuracy 5.6 pts","Trace multilingual agent failures to request-to-plan gaps","Fix five plan-grounding errors to gain 5.6 pts on multilingual GAIA","Systematic plan-grounding failures hinder multilingual agents","Actionable taxonomy for multilingual multi-agent planning failures"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The experiments compare TART against direct planning, not against a control that gets an equally structured summary with arbitrary fields, so the specific taxonomy content—not the mere act of structured restatement—is the assumed cause of the gains.","fun_headline_variants_meta":{"raw":{"variants":["Planning-failure taxonomy lifts multilingual agent accuracy 5.6 pts","Trace multilingual agent failures to request-to-plan gaps","Fix five plan-grounding errors to gain 5.6 pts on multilingual GAIA","Systematic plan-grounding failures hinder multilingual agents","Actionable taxonomy for multilingual multi-agent planning failures"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000541,"raw_usage":{"total_tokens":2390,"prompt_tokens":668,"completion_tokens":1722,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":412,"completion_tokens_details":{"reasoning_tokens":1639}},"tokens_in":412,"tokens_out":1722,"duration_ms":15661,"temperature":1.0,"reasoning_tokens":1639,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:46:42.050624+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the GAIA-MAPS evaluation with a control condition in which the converter produces a JSON summary with generic fields instead of the five taxonomy fields; if that control matches TART's gains, the taxonomy's specific content is not the active mechanism. The paper's ablation varies which taxonomy fields are included but never removes the converter, so this control would settle whether the diagnosis is actionable or the structure alone helps.","supporting_citations":[],"review_version":1}