{"id":"0b01c2b2-4227-4a5f-aa3f-50a044042c76","arxiv_id":"2508.12981","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-agent LLM team with a shared notebook and an orchestrator reaches a 25% pass rate on TravelPlanner, up from 7.5% for a single agent.","lead":"This paper tests whether a shared notebook and an orchestrator agent improve how well LLM teams plan trips with many constraints. The combined system boosts TravelPlanner's pass rate from 7.5% for a single agent to 25%, an early sign that coordination structure matters as much as prompting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract lacks variance/significance tests; the 17.5-point gap and 13.5% sub-area gain could stem from run-to-run noise or post-hoc sub-area selection.","rationale":"The reader's verdict is UNVERDICTED because the full text is unavailable. My stress-test focused on the central numerical claim that is available in the abstract. The most fragile link is the reliability of the 17.5-point improvement: if it is a single-run artifact, the paper's main contribution evaporates. This is not a claim of manipulation; it is a standard demand in empirical LLM work for repeated runs and paired significance testing. I agree with the reader's weakest assumption, which highlights run-to-run variance and post-hoc sub-area selection. The concrete test would settle whether the gap is genuine. Since even a positive result would need full-text details, the verdict remains UNVERDICTED/unchanged; the concern does not move the verdict to a stronger rejection, but it does mean the paper cannot be accepted on the abstract alone.","tokens_in":769,"tokens_out":5383,"duration_ms":61575,"concrete_test":"Obtain from the authors the full TravelPlanner evaluation logs: per-query pass/fail labels for each condition, across at least 3 independent seeds (e.g., temperature 0, 0.7, 1.0) and a fixed set of 180 held-out queries. Compute Wilson 95% confidence intervals for the single-agent and combined pass rates and a paired McNemar test for the difference. Require a pre-registered (or at least fixed in advance) definition of 'focused sub-areas' before running the orchestrator; if sub-areas were selected post hoc, repeat evaluation on a separate hold-out split. If the combined pass rate is not significantly better than the baseline at p<0.05, or if the sub-area improvement does not replicate, the headline claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TravelPlanner is a fixed 180-query benchmark; with binary per-query pass/fail, the difference between 7.5% and 25% across paired conditions is statistically testable but only if the authors report the underlying per-query outcomes and repeat runs. The abstract provides only point estimates. LLM sampling temperature, API nondeterminism, and prompt-order effects make single-run pass rates unstable; a low baseline (7.5%) could be a run that happened to perform poorly, inflating the relative gain. More importantly, the orchestrator's 'up to 13.5% error reduction within focused sub-areas' is an upper bound over an unspecified set of sub-areas; if those sub-areas were chosen by examining baseline errors, the 25% combined pass rate is partially a product of benchmark overfitting. These are not internal contradictions but missing evidence, and they are load-bearing because the abstract's causal conclusion ('structured information sharing and reflective orchestration as key components') rests entirely on this gap being reproducible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs an LLM-based multi-agent system for the TravelPlanner travel-planning benchmark. It evaluates two mechanisms: a shared notebook for information sharing and an orchestrator agent for coordination. The abstract reports that the notebook reduces hallucinated-detail errors by 18%, the orchestrator reduces errors by up to 13.5% within focused sub-areas, and the combined system achieves a 25% final pass rate versus a 7.5% single-agent baseline, a 17.5-point absolute improvement. The paper interprets these results as evidence that structured information sharing and reflective orchestration are key components for long-horizon planning with LLMs.","tokens_in":1013,"tokens_out":1613,"duration_ms":18472,"significance":"If the reported effects are real and reproducible, the paper provides a useful empirical contribution to the growing literature on LLM multi-agent coordination, showing that explicit shared memory and a reflective controller can improve performance on a concrete, externally anchored benchmark. The use of TravelPlanner as an external benchmark is a strength: evaluation is not self-referential. However, significance is currently bounded by the absence of any statistical grounding for the headline numbers and by the unspecified selection of the orchestrator's 'focused sub-areas.' The claimed 17.5-point improvement is the central empirical finding, and its credibility depends on evidence that is not presented in the abstract.","major_comments":[{"comment":"The abstract reports point estimates (18%, 13.5%, 25%, 7.5%) without error bars, confidence intervals, number of runs, or significance tests. TravelPlanner is a fixed set of 180 queries with binary per-query pass/fail outcomes, so the difference between 7.5% and 25% is statistically testable, but only if the authors report per-query outcomes and repeated runs under LLM nondeterminism. As written, the headline 17.5-point gain could reflect run-to-run variance, especially given the low baseline. This is load-bearing for the causal conclusion and needs to be supported with variance-aware reporting.","section":"Abstract"},{"comment":"The 'up to 13.5%' error reduction is claimed 'within focused sub-areas,' but the abstract does not state how those sub-areas were selected. If the sub-areas were chosen after observing where the baseline or notebook-only system failed, the number is an upper bound on a post-hoc selection and the combined 25% pass rate may partially result from benchmark overfitting. The authors must specify the selection rule, ideally pre-registered, and report performance across all sub-areas, not only the focused ones.","section":"Abstract"},{"comment":"The single-agent baseline pass rate of 7.5% is unusually low for TravelPlanner in the existing literature (where reports vary, but often higher). A single unlucky baseline run would inflate the relative improvement. The manuscript should report the distribution of baseline pass rates over multiple runs or seeds and confirm that the 7.5% figure is not an outlier.","section":"Abstract"}],"minor_comments":[{"comment":"Clarify how 'hallucinated details' errors are operationalized and counted; this is central to the 18% reduction claim.","section":"Abstract"},{"comment":"Provide precise definitions of the 'notebook' and 'orchestrator' mechanisms; the abstract gives only a high-level description.","section":"Abstract"},{"comment":"Indicate the number of runs and sampling temperature used for each condition; without this, even point estimates are hard to interpret.","section":"Abstract"},{"comment":"Consider reporting performance on the full TravelPlanner test set, not only focused sub-areas, to contextualize the combined pass rate.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"This review is based solely on the abstract, as the full text was not provided. The central claim is plausible and the benchmark choice is appropriate, but the missing statistical evidence and the unspecified sub-area selection are load-bearing and cannot be resolved by presentation fixes alone. I recommend major revision with a request for the full experimental details. I would be happy to re-review once the complete manuscript, including run-level data and sub-area selection criteria, is available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real empirical contribution in the making, but the abstract alone doesn't let us check the load-bearing numbers. The headline—25% vs 7.5% pass rate with a shared notebook plus orchestrator—is the kind of concrete result that would be useful for the LLM planning subfield. The mechanisms are not new (shared notebooks and orchestrators are established), but the specific combination and the TravelPlanner measurements are new. That is a legitimate extension, not a field re-organizer.\n\nWhat the paper does well: it anchors to a public benchmark, reports an absolute improvement, and the design (two mechanisms, evaluated separately and together) is clean. The 18% reduction in hallucinated details is a specific, quantifiable effect.\n\nWhere I get worried: all numbers are point estimates with no error bars, run counts, or significance tests. With a binary per-query pass/fail on 180 queries, the 7.5% vs 25% gap is large, but LLM sampling noise could easily move a 7.5% baseline. The 'up to 13.5% error reduction within focused sub-areas' is the biggest red flag—'up to' plus unspecified sub-area selection could mean the authors picked the best-looking slice after seeing the data. They need to state how the sub-areas were chosen and report the underlying query-level results. The abstract also cites no prior work, which makes it hard to see what is actually new relative to known orchestrator/notebook patterns.\n\nThe central claim is not circular—it's an external benchmark—but the benchmark overfitting risk is real if the focus areas were tuned on baseline errors. This is missing evidence, not internal contradiction.\n\nBottom line: the paper deserves a serious referee. I'd want the full method, the per-query outcomes, and a pre-registered or at least clearly defined sub-area selection. If those hold, this is a useful data point. For now, treat it as promising but unverified.\n\nRecommendation: send to peer review. And I'd bring the full version to reading group when it's available.","headline":"A plausible, useful empirical result on TravelPlanner that is hard to evaluate from the abstract alone—the missing statistics and sub-area selection are the main risks.","tokens_in":1490,"tokens_out":1660,"would_cite":false,"duration_ms":17378,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding a shared notebook and an orchestrator agent to an LLM multi-agent system lifts a travel-planning pass rate from 7.5% to 25%.","keywords":["multi-agent systems","large language models","travel planning","information sharing","orchestrator agent","long-horizon planning","constraint satisfaction","TravelPlanner"],"falsifier":"Run the single-agent baseline and the notebook-plus-orchestrator system on a held-out set of travel-planning constraints with multiple random seeds, with the orchestrator's focus rules fixed before seeing baseline errors. If the 17.5-percentage-point pass-rate gap does not reproduce, or if removing the notebook's stored details leaves hallucination-error counts unchanged, the claimed causal roles of the two mechanisms fail.","tokens_in":657,"feed_emoji":"🗺️","tokens_out":5022,"duration_ms":43246,"temperature":0.7,"pith_summary":"This paper asks whether two structural additions—a shared notebook and an orchestrator agent—make an LLM-based multi-agent system better at long-horizon planning that must satisfy many interacting constraints. It tests this on a travel-planning benchmark, where a plan must stitch together flights, hotels, restaurants, and activities while respecting budget, dates, and preferences. The paper reports that the notebook alone reduces errors due to hallucinated details by 18%, the orchestrator alone further reduces errors by up to 13.5% in focused sub-areas, and the two together raise the final pass rate from 7.5% to 25%. A sympathetic reader would take this as evidence that explicit information sharing and reflective coordination are key components for making LLM agent teams reliable on constraint-heavy tasks.","feed_headline":"Notebook plus coordinator lifts AI planner pass rate from 7.5% to 25%","feed_subtitle":"A shared notebook cuts hallucinated details by 18%; a coordinator adds focus, roughly tripling final success rate.","key_machinery":"The paper's central mechanism is a two-part architecture. A shared notebook is an external structured record of plan details that agents consult and update, reducing reliance on conversational memory and preventing hallucinated facts. An orchestrator agent is a reflective coordinator that evaluates the plan's sub-areas during free-form agent conversation and directs agents to focus on the weakest parts. Together they target distinct failure modes: the notebook keeps details faithful, and the orchestrator keeps collective effort aligned with the constraints that still need work.","core_discovery":"The central claim is that combining a shared notebook with an orchestrator agent produces a 25% final pass rate on the TravelPlanner benchmark, a 17.5-percentage-point absolute improvement over a single-agent baseline's 7.5%. The notebook functions as external structured memory, anchoring agents in previously agreed details and cutting hallucination-driven errors by 18%. The orchestrator watches the free-form conversation among agents, identifies sub-areas of the plan needing attention, and guides focused refinement, cutting errors by up to 13.5% within those sub-areas. The two mechanisms attack complementary failure modes—detail grounding and coordination focus—and are presented as key comp","pith_inferences":["A natural next experiment the abstract does not report: pre-register the orchestrator's focus rules on one set of constraints and test on unseen constraint families; if the 13.5% sub-area gain persists, the effect is a general coordination mechanism rather than a consequence of chosen sub-areas.","If these gains transfer to other multi-constraint planning domains—trip scheduling, logistics routing, event planning—then adding an external shared memory and a reflective coordinator could become a standard default for LLM agent teams, independent of model choice.","The 18% reduction in hallucinated details suggests that external structured memory may contribute more to long-horizon plan faithfulness than increasing model scale or prompt length, since it directly addresses where plans lose contact with the user's stated constraints."],"forward_implications":["A shared notebook as external structured memory cuts hallucinated-detail errors by 18% in LLM multi-agent travel planning.","An orchestrator that selects sub-areas for focused conversation further reduces errors by up to 13.5% within those areas.","Combining both mechanisms yields a 25% final pass rate on TravelPlanner, a 17.5-percentage-point absolute improvement over the 7.5% single-agent baseline.","The two mechanisms are complementary—one addresses factual grounding, the other addresses coordination focus—so systems aiming for reliable long-horizon planning should include both rather than either alone.","Success on TravelPlanner, a representative multi-constraint planning task, suggests these design choices are plausible components for LLM multi-agent systems generally."],"supporting_citations":[],"fun_headline_variants":["Notebook + orchestrator triples AI planner success rate","AI planner pass rate triples with shared notebook and coordinator","Notebook cuts hallucinated details 18%; orchestrator adds focus","25% pass rate for AI planners: notebook and coordinator key","Shared notebook and orchestrator: AI planning success up 3x"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the TravelPlanner pass rate is a faithful, stable measure of long-horizon multi-constraint planning quality, so that the 7.5%-versus-25% gap reflects genuine capability rather than benchmark noise or run-to-run variance; likewise, the orchestrator's 13.5% gain is only meaningful if its focus sub-areas were not chosen after observing where the baseline failed.","fun_headline_variants_meta":{"raw":{"variants":["Notebook + orchestrator triples AI planner success rate","AI planner pass rate triples with shared notebook and coordinator","Notebook cuts hallucinated details 18%; orchestrator adds focus","25% pass rate for AI planners: notebook and coordinator key","Shared notebook and orchestrator: AI planning success up 3x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000907,"raw_usage":{"total_tokens":3725,"prompt_tokens":724,"completion_tokens":3001,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":468,"completion_tokens_details":{"reasoning_tokens":2915}},"tokens_in":468,"tokens_out":3001,"duration_ms":28872,"temperature":1.0,"reasoning_tokens":2915,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T19:10:12.491875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the single-agent baseline and the notebook-plus-orchestrator system on a held-out set of travel-planning constraints with multiple random seeds, with the orchestrator's focus rules fixed before seeing baseline errors. If the 17.5-percentage-point pass-rate gap does not reproduce, or if removing the notebook's stored details leaves hallucination-error counts unchanged, the claimed causal roles of the two mechanisms fail.","supporting_citations":[],"review_version":1}