REVIEW 3 major objections 4 minor
Analyzing Information Sharing and Coordination in Multi-Agent Planning
T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that adding a shared notebook and an orchestrator agent to an LLM multi-agent system lifts a travel-planning pass rate from 7.5% to 25%.
desk verdict A plausible, useful empirical result on TravelPlanner that is hard to evaluate from the abstract alone—the missing statistics and sub-area selection are the main risks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The paper's central mechanism is a two-part architecture. A shared notebook is an external structured record of plan details that agents consult and update, reducing reliance on conversational memory and preventing hallucinated facts. An orchestrator agent is a reflective coordinator that evaluates the plan's sub-areas during free-form agent conversation and directs agents to focus on the weakest parts. Together they target distinct failure modes: the notebook keeps details faithful, and the orchestrator keeps collective effort aligned with the constraints that still need work.
What would settle it
Run the single-agent baseline and the notebook-plus-orchestrator system on a held-out set of travel-planning constraints with multiple random seeds, with the orchestrator's focus rules fixed before seeing baseline errors. If the 17.5-percentage-point pass-rate gap does not reproduce, or if removing the notebook's stored details leaves hallucination-error counts unchanged, the claimed causal roles of the two mechanisms fail.
Extended reading notes
Core claim
The central claim is that combining a shared notebook with an orchestrator agent produces a 25% final pass rate on the TravelPlanner benchmark, a 17.5-percentage-point absolute improvement over a single-agent baseline's 7.5%. The notebook functions as external structured memory, anchoring agents in previously agreed details and cutting hallucination-driven errors by 18%. The orchestrator watches the free-form conversation among agents, identifies sub-areas of the plan needing attention, and guides focused refinement, cutting errors by up to 13.5% within those sub-areas. The two mechanisms attack complementary failure modes—detail grounding and coordination focus—and are presented as key comp
Load-bearing premise
The load-bearing premise is that the TravelPlanner pass rate is a faithful, stable measure of long-horizon multi-constraint planning quality, so that the 7.5%-versus-25% gap reflects genuine capability rather than benchmark noise or run-to-run variance; likewise, the orchestrator's 13.5% gain is only meaningful if its focus sub-areas were not chosen after observing where the baseline failed.
Editorial extensions
If this is right
- A shared notebook as external structured memory cuts hallucinated-detail errors by 18% in LLM multi-agent travel planning.
- An orchestrator that selects sub-areas for focused conversation further reduces errors by up to 13.5% within those areas.
- Combining both mechanisms yields a 25% final pass rate on TravelPlanner, a 17.5-percentage-point absolute improvement over the 7.5% single-agent baseline.
- The two mechanisms are complementary—one addresses factual grounding, the other addresses coordination focus—so systems aiming for reliable long-horizon planning should include both rather than either alone.
- Success on TravelPlanner, a representative multi-constraint planning task, suggests these design choices are plausible components for LLM multi-agent systems generally.
Reading between the lines
- A natural next experiment the abstract does not report: pre-register the orchestrator's focus rules on one set of constraints and test on unseen constraint families; if the 13.5% sub-area gain persists, the effect is a general coordination mechanism rather than a consequence of chosen sub-areas.
- If these gains transfer to other multi-constraint planning domains—trip scheduling, logistics routing, event planning—then adding an external shared memory and a reflective coordinator could become a standard default for LLM agent teams, independent of model choice.
- The 18% reduction in hallucinated details suggests that external structured memory may contribute more to long-horizon plan faithfulness than increasing model scale or prompt length, since it directly addresses where plans lose contact with the user's stated constraints.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper constructs an LLM-based multi-agent system for the TravelPlanner travel-planning benchmark. It evaluates two mechanisms: a shared notebook for information sharing and an orchestrator agent for coordination. The abstract reports that the notebook reduces hallucinated-detail errors by 18%, the orchestrator reduces errors by up to 13.5% within focused sub-areas, and the combined system achieves a 25% final pass rate versus a 7.5% single-agent baseline, a 17.5-point absolute improvement. The paper interprets these results as evidence that structured information sharing and reflective orchestration are key components for long-horizon planning with LLMs.
Significance. If the reported effects are real and reproducible, the paper provides a useful empirical contribution to the growing literature on LLM multi-agent coordination, showing that explicit shared memory and a reflective controller can improve performance on a concrete, externally anchored benchmark. The use of TravelPlanner as an external benchmark is a strength: evaluation is not self-referential. However, significance is currently bounded by the absence of any statistical grounding for the headline numbers and by the unspecified selection of the orchestrator's 'focused sub-areas.' The claimed 17.5-point improvement is the central empirical finding, and its credibility depends on evidence that is not presented in the abstract.
major comments (3)
- [Abstract] The abstract reports point estimates (18%, 13.5%, 25%, 7.5%) without error bars, confidence intervals, number of runs, or significance tests. TravelPlanner is a fixed set of 180 queries with binary per-query pass/fail outcomes, so the difference between 7.5% and 25% is statistically testable, but only if the authors report per-query outcomes and repeated runs under LLM nondeterminism. As written, the headline 17.5-point gain could reflect run-to-run variance, especially given the low baseline. This is load-bearing for the causal conclusion and needs to be supported with variance-aware reporting.
- [Abstract] The 'up to 13.5%' error reduction is claimed 'within focused sub-areas,' but the abstract does not state how those sub-areas were selected. If the sub-areas were chosen after observing where the baseline or notebook-only system failed, the number is an upper bound on a post-hoc selection and the combined 25% pass rate may partially result from benchmark overfitting. The authors must specify the selection rule, ideally pre-registered, and report performance across all sub-areas, not only the focused ones.
- [Abstract] The single-agent baseline pass rate of 7.5% is unusually low for TravelPlanner in the existing literature (where reports vary, but often higher). A single unlucky baseline run would inflate the relative improvement. The manuscript should report the distribution of baseline pass rates over multiple runs or seeds and confirm that the 7.5% figure is not an outlier.
minor comments (4)
- [Abstract] Clarify how 'hallucinated details' errors are operationalized and counted; this is central to the 18% reduction claim.
- [Abstract] Provide precise definitions of the 'notebook' and 'orchestrator' mechanisms; the abstract gives only a high-level description.
- [Abstract] Indicate the number of runs and sampling temperature used for each condition; without this, even point estimates are hard to interpret.
- [Abstract] Consider reporting performance on the full TravelPlanner test set, not only focused sub-areas, to contextualize the combined pass rate.
Circularity Check
No circularity: all claims are empirical outcomes on an external benchmark, with no fitted parameters or self-cited premises.
full rationale
The abstract reports direct measurements on the external TravelPlanner benchmark. The 18% notebook-related error reduction, the 'up to 13.5%' orchestrator error reduction, and the combined 25% pass rate are stated as observed results, not as quantities derived from definitions or from fitted inputs. There is no equation in which the output is defined in terms of the input, no parameter fitted to a subset of the reported outcome and then renamed as a prediction, and no citation to prior work (self or otherwise) that supplies a load-bearing premise. The only possible concern is that the 'focused sub-areas' could have been chosen after inspecting baseline errors, making the 'up to 13.5%' an optimistic selection rather than a fixed hypothesis. However, this is a statistical/multiple-comparisons concern about post-hoc selection, not a circular derivation; the abstract does not state that those sub-areas were selected using the baseline outcomes, and the combined pass rate is still an independent external measurement. Under the hard rule requiring a quotable reduction to inputs, no circular step is present. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (1)
- orchestrator focus sub-areas =
not reported
assumptions (3)
- domain assumption TravelPlanner's pass rate is a valid, representative measure of long-horizon multi-constraint planning quality
- domain assumption The single-agent configuration is the appropriate baseline for measuring the mechanisms' contribution
- domain assumption The orchestrator's interventions are the causal driver of the observed error reduction
invented entities (2)
-
shared notebook
independent evidence
-
orchestrator agent
independent evidence
Cite this review
Pith. "Pith review of Analyzing Information Sharing and Coordination in Multi-Agent Planning." pith.science (2026). https://pith.science/paper/MST6GACJ
@misc{pith2026250812981,
author = {Pith},
title = {Pith review of: Analyzing Information Sharing and Coordination in Multi-Agent Planning},
year = {2026},
howpublished = {\url{https://pith.science/paper/MST6GACJ}},
note = {Machine review of arXiv:2508.12981}
}
read the original abstract
Multi-agent systems (MASs) have pushed the boundaries of large language model (LLM) agents in domains such as web research and software engineering. However, long-horizon, multi-constraint planning tasks involve conditioning on detailed information and satisfying complex interdependent constraints, which can pose a challenge for these systems. In this study, we construct an LLM-based MAS for a travel planning task which is representative of these challenges. We evaluate the impact of a notebook to facilitate information sharing, and evaluate an orchestrator agent to improve coordination in free form conversation between agents. We find that the notebook reduces errors due to hallucinated details by 18%, while an orchestrator directs the MAS to focus on and further reduce errors by up to 13.5% within focused sub-areas. Combining both mechanisms achieves a 25% final pass rate on the TravelPlanner benchmark, a 17.5% absolute improvement over the single-agent baseline's 7.5% pass rate. These results highlight the potential of structured information sharing and reflective orchestration as key components in MASs for long horizon planning with LLMs.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.