{"id":"5613eec1-5823-46b6-885a-7168f8194dff","arxiv_id":"2412.13520","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A role-based multi-agent framework with a monitor that triggers re-planning is reported to outperform other LLM agent systems on two QA benchmarks, but no code, data, or error bars are provided.","lead":"ROMAS, a role-based multi-agent system built on the DB-GPT framework, assigns planner, monitor, and worker roles that plan, self-reflect, and re-plan database analytics tasks. The paper reports higher success rates on financial QA and multi-hop reasoning benchmarks than existing agent frameworks, but ships no code or data to check the numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ROMAS's headline margins over AutoAgents are 8/100 and 6/100 questions with no confidence intervals or seed variation; subset selection or noise could fully explain the reported superiority.","rationale":"I read the paper as a system/engineering claim: ROMAS is a role-based multi-agent system on DB-GPT that beats AutoAgents and other baselines on two benchmarks. The most load-bearing condition for that claim is that the measured superiority is real and not an artifact of the evaluation. That condition is least secure because the evaluation uses 100 self-selected cases per dataset, reports no uncertainty, and transforms the FAMMA inputs by converting table images into tabular text. The reader emphasized the undisclosed error tree in Appendix D; I agree that is a gap, but even a perfect error tree would not rescue the headline if the comparison itself is not statistically grounded. Conversely, if the full-benchmark or repeated-draw check reproduces the margin, the central claim survives despite the error-tree opacity. The paper has useful engineering content (deployment in DB-GPT, plausible ablation directions), but as a claim of superiority over AutoAgents the evidence is insufficient. I therefore concur with the reader's REJECT verdict and would not change it.","tokens_in":16317,"tokens_out":3654,"duration_ms":33637,"concrete_test":"Ask the authors to release the exact 100 case IDs and the table-to-text conversion script for FAMMA, plus the AutoAgents configuration. Then: (1) run ROMAS and AutoAgents on the full FAMMA benchmark (or on five fresh 100-case random draws satisfying the stated filter) with temperature 0; (2) compute paired bootstrap 95% confidence intervals for the success-rate difference and a McNemar test on the paired cases. If the confidence interval includes zero, or the margin on the full set is substantially below the reported 8.23 points, the superiority claim is not established. As a secondary check, rerun the Table 4 no-monitor ablation on the full benchmark to confirm the 22.66-point drop.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is 'ROMAS outperforms the others' (Table 3), but every number in that table is a point estimate on 100 self-selected questions. On FAMMA the margin over AutoAgents is 8.23 points; on HotpotQA it is 6.25 points — 8 and 6 correct answers out of 100. No confidence interval, bootstrap, McNemar test, or repeated-seed result is reported. The FAMMA subset is not a random sample: §4 says the authors selected cases containing both text and table images with standard options and then converted the table images to tabular format. This filter-plus-conversion changes the task (removing the multimodal component) and could differentially favor the ROMAS pipeline, whose extractor/retriever is designed around parsed tables. If even a few of the 8/6 discriminating questions are artifacts of subset choice or of the table-conversion step, the headline ranking reverses. Because the ablation study (Table 4) is run on the same subset, the monitor's 22.66-point importance is also not disentangled from selection effects. The paper also does not state the AutoAgents prompt or configuration used for the baseline. The claim therefore requires that the ranking is robust on the full benchmark or on fresh independent draws with statistical testing; the paper provides no such evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes ROMAS, a role-based multi-agent system deployed inside the DB-GPT framework. It organizes agents into a planner, a monitor, and workers; the planner generates an agent team and task lists, workers execute and self-reflect, and the monitor classifies errors by searching a predefined error tree and either repairs pipeline errors or triggers re-planning through a 'gap narrow' rule. The empirical section compares ROMAS with LLM, single-agent, and multi-agent baselines on 100-sample subsets of FAMMA and HotpotQA, reports ablations, and compares DB-GPT with LangChain and AgentScope. The paper claims ROMAS achieves 81.68% success on FAMMA and 85.24% on HotpotQA, outperforming all baselines, and that DB-GPT enables significantly less code.","tokens_in":16588,"tokens_out":7466,"duration_ms":67146,"significance":"If the reported results were robust, ROMAS would be a useful instance of a role-based, self-monitoring multi-agent architecture for database-backed QA, and the deployment in a real open-source framework (DB-GPT) is a concrete strength. The separation of planner/monitor/workers and the gap-narrow replanning mechanism are clearly described, and the ablation design at least targets the main components. The central weakness is that every quantitative claim rests on 100-case self-selected subsets without uncertainty quantification, and the baselines and components are not fully specified; the empirical contribution is therefore not yet reproducible. The paper would be strengthened substantially by code/data release and full-benchmark or properly sampled evaluation with statistical tests.","major_comments":[{"comment":"The main claim that ROMAS 'outperforms the others' is supported only by point estimates on 100 self-selected samples per dataset. The margin over AutoAgents is 8.23 points on FAMMA and 6.25 points on HotpotQA, i.e. roughly 8 and 6 questions out of 100, and no confidence interval, bootstrap, McNemar test, or repeated-seed result is reported. The FAMMA subset is further filtered to cases containing both text and table images and then converted from images to tabular format, which removes the multimodal component and may change task difficulty. To support the central claim, report results on the full benchmarks or on a documented random sample with uncertainty quantification, and report baselines under identical conditions.","section":"§4 Datasets / Table 3"},{"comment":"The monitor's error classification relies on a predefined error tree said to be built from 'empirical data', but the paper does not describe the tree's nodes, construction, coverage, or validation; Fig. 7 is only a coarse schematic. Because Table 4 attributes a 22.66-point FAMMA drop to removing the monitor, an underspecified error tree is a load-bearing component. Please provide the full tree (list of nodes/leaves), examples of error instances, and an analysis of classification accuracy/coverage.","section":"§3.2 / Appendix D"},{"comment":"The baseline configurations are not reproducible. The paper does not state prompt templates, retry limits, temperature, tool access, or database connection details for CoT, ToT, ReAct, Generative Agents, and AutoAgents. Since ROMAS operates inside DB-GPT with its own tools, differences in tool availability and retry budget could explain the reported margins. Provide exact code/configurations and, ideally, run baselines within the same DB-GPT environment.","section":"§4 Setup / Analysis I"},{"comment":"The ablation study is performed on the same 100-sample self-selected subsets and reports single runs without variance. In particular, the largest ablation drop (22.66 points on FAMMA) is not accompanied by repeated seeds or error bars, so it is not possible to separate the effect of the component from subset-selection effects. Please add repeated runs, confidence intervals, and ideally ablations on the full benchmark.","section":"§4 Analysis II / Table 4"},{"comment":"The DB-GPT effectiveness comparison is conducted by the authors of DB-GPT and ROMAS, using 'code volume (number of rows)' as a metric. The table does not report which features are implemented, the exact counting convention, or the code repositories, and no significance testing is provided. This weakens the low-code-development claim; please make the implementations public and specify the feature set per framework, or use an independent implementer.","section":"§4 Analysis III / Table 5"}],"minor_comments":[{"comment":"There are typos such as 'versatil', 'specailized', 'Re-plannig', and 'Cleanning'; these should be corrected.","section":"Throughout"},{"comment":"The sentence refers to 'table 4' when the DB-GPT comparison is in Table 5.","section":"§4 Analysis III"},{"comment":"The caption and figure for the error tree do not show the actual nodes or leaf definitions; please include a legible tree or a tabular listing.","section":"Appendix D"},{"comment":"The description of the LLM and human evaluation does not state which model served as judge, how the ten dimensions are aggregated, or how expert disagreement was handled; an inter-annotator agreement statistic would help.","section":"Appendix C"},{"comment":"The loop over differences and rules is ambiguous, especially after a difference is regenerated; please clarify with explicit continue/break conditions.","section":"Algorithm 1"},{"comment":"The checkmarks in Tables 1 and 2 would benefit from a precise legend defining each capability and the evidence used for each mark.","section":"Tables 1-2"}],"recommendation":"major_revision","confidential_remarks":"I would advise the editor that this is a system paper whose contribution is mainly empirical, but the evidence currently falls below the standard of the journal. The DB-GPT comparison is authored by the framework's developers, so independent verification or released code/repositories is essential. The shortcomings are addressable in principle, but the revision would need to be substantial rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before you cite it as evidence that role-based multi-agent systems beat generic LLM baselines. It's a system description with a plausible architecture—planner, monitor, workers, plus an error-tree classification and a 'gap narrow' re-planning rule—but the evaluation is too thin to support the headline that ROMAS 'outperforms the others.'\n\nWhat's genuinely useful here: the decomposition is clean and the authors situate it against AutoAgents, MetaGPT, and Generative Agents. They also ship the framework inside DB-GPT, which is real engineering: the comparison of implementation effort (1500 vs 1800 vs 2500 lines) is a nice concrete datapoint. The ablation study, even on the same 100-question slice, consistently shows the monitor mechanism is the biggest lever, which gives the design some internal coherence.\n\nThe soft spots are where the reader's rejection is on point. Every success rate in Table 3 is a point estimate on 100 self-selected questions. The margins over AutoAgents are 8.23 points on FAMMA and 6.25 on HotpotQA—eight and six questions. No confidence intervals, no repeated seeds, no significance test. The FAMMA subset is filtered to cases with both text and table images, then the table images are converted to tabular format; that changes the task and could favor ROMAS's extractor/retriever design. The monitor's load-bearing error tree is shown as a figure with no node details, and the 'empirical data' behind it is undisclosed. Without that specification, the mechanism is not reproducible. There are also minor internal inconsistencies (Analysis III refers to Table 4 instead of Table 5; a couple of typos) that suggest haste.\n\nI agree with the reader's verdict that the current evidence doesn't demonstrate superiority. But I'd stop short of calling this a weak paper. The architecture is reasonable, the deployment is real, and the ablation gives a coherent story. The right fix is to run the full benchmark (or at least a pre-registered random subsample with McNemar or bootstrap tests), release prompts and code, and spell out the error tree. That's a major revision, not a desk reject.\n\nRecommendation: send to peer review — the work deserves referee time, but expect the reviewers to send it back for substantial empirical additions. For your own reading group, it's a good example of how easy it is to over-read a 100-sample comparison.","headline":"Plausible agent framework with a thin 100-sample evaluation; the monitor mechanism is interesting but the claimed superiority is not established.","tokens_in":17135,"tokens_out":2740,"would_cite":false,"duration_ms":24535,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A role-based multi-agent system with a global monitor reports higher accuracy than prior LLM agent systems on financial and multi-hop reasoning benchmarks.","keywords":["multi-agent systems","LLM agents","role-based cooperation","self-monitoring","self-planning","database analytics","error tree search","low-code development"],"falsifier":"Take a random sample of failure logs from the two benchmarks, have independent raters label each error's branch, and compare their labels with the monitor's classifications; if agreement is low, the error tree is not doing the claimed work. A sharper test is to inject a failure type outside the paper's two branches, such as a tool-permission or prompt-security error, and observe whether the monitor can still produce a repair.","tokens_in":16071,"feed_emoji":"🤖","tokens_out":11122,"duration_ms":88103,"temperature":0.7,"pith_summary":"This paper argues that LLM-powered multi-agent systems for data analysis fail mainly because they cannot correct an executed plan, not because they cannot make one. It introduces ROMAS, a role-based system in which a planner builds an agent team and task list, workers execute, and a monitor observes the global state, classifies errors, and either repairs them directly or sends the planner back to re-plan. On 100-case subsets of a financial multimodal QA benchmark and a multi-hop reasoning benchmark, ROMAS reports success rates of 81.68% and 85.24%, above the strongest role-based comparison system. If the result holds, the practical lesson is that an explicit monitoring and re-planning role is what makes a multi-agent system robust on real data tasks.","feed_headline":"Adding a monitor role to agent systems lifts accuracy past 85%","feed_subtitle":"A planner-monitor-worker design scores 81.7% on financial QA and 85.2% on multi-hop reasoning.","key_machinery":"The load-bearing mechanism is the monitor's error tree search and the re-planning loop. The monitor uses a predefined error tree, built from empirical data, with two top-level branches: pipeline errors and logical (agent-team generation) errors; it searches from the root to a leaf node by similarity to classify what went wrong. Pipeline errors receive direct correction instructions, while agent-team errors send global information and recommendations back to the planner, which regenerates a strategy under the gap-narrow rule, a constraint that changes the previous plan as little as possible while fixing the observed failures.","core_discovery":"The central claim is that the bottleneck in LLM multi-agent data analytics is the absence of an execution-time correction channel. ROMAS instantiates a planner, a monitor, and a set of workers, then adds a re-planning phase: workers attempt self-reflection first, broadcast their global state to the monitor on persistent failure, and the monitor classifies each error as a pipeline error, which it fixes directly, or an agent-team generation error, which triggers the planner to rebuild the strategy. The paper reports that removing the monitor causes the largest success-rate drop on the financial benchmark, 22.66 percentage points, and that the full system reaches 81.68% on FAMMA and 85.24% on HotpotQA.","pith_inferences":["If the monitor's error tree is the load-bearing component, a natural extension is to learn the tree from failure logs rather than fixing it by hand and to measure how often the monitor's classification agrees with human raters.","The gap-narrow rule states a general principle for LLM agents: constrain revisions to the smallest edit that fixes the observed failure, a principle that should transfer to code repair and other self-correcting settings.","The reported numbers come from 100-case subsets, so scaling to the full benchmarks would test whether the margins survive broader sampling.","A production side effect of an explicit monitor role is auditability: each correction can be attributed to a monitor decision rather than hidden inside a planner prompt."],"forward_implications":["Plan-only multi-agent architectures are insufficient: the ablation attributes the largest single drop on the finance benchmark, 22.66 percentage points, to removing the monitor.","Error correction is tiered by cost: workers self-correct, the monitor fixes pipeline errors directly, and only agent-team generation errors trigger a full re-plan.","The gap-narrow rule keeps re-planning cheap by reusing the old strategy, which is why the re-planning phase can run within a bounded number of retries.","The component that matters most depends on the task: the finance benchmark loses most from removing the monitor, while the multi-hop reasoning benchmark loses most from removing memory and self-reflection.","A database-focused low-code framework can implement the same system with less development code and lower average response time than general-purpose agent frameworks in the paper's comparison."],"supporting_citations":[{"why":"Supplies the DB-GPT framework that ROMAS uses for low-code development and database analytics.","marker":"Xue et al., 2023a, 2024b"},{"why":"Supplies the role-based agent-generation baseline whose scores ROMAS is compared against.","marker":"Chen et al., 2024"},{"why":"Supplies the reflective-agent baseline in the state-of-the-art MAS comparison.","marker":"Park et al., 2023"},{"why":"Supplies the task-oriented multi-agent baseline lacking a global monitor in the comparison.","marker":"Hong et al., 2024"},{"why":"Supplies the FAMMA financial QA benchmark used for the main evaluation.","marker":"Xue et al., 2024a"},{"why":"Supplies the HotpotQA multi-hop reasoning benchmark used for general evaluation.","marker":"Yang et al., 2018"},{"why":"Supplies the general-purpose agent framework baseline in the code-volume comparison.","marker":"Chase, 2022"},{"why":"Supplies the alternative multi-agent platform baseline in the framework comparison.","marker":"Gao et al., 2024"},{"why":"Supplies the gap-narrow, minimum-difference principle used by the re-planning phase.","marker":"Li et al., 2024b"},{"why":"Supplies the minimum-adjustment feedback principle that motivates the gap-narrow rule.","marker":"Zhang et al., 2020"}],"fun_headline_variants":["Monitor role is key to 85.2% multi-agent accuracy","Without a monitor, agent accuracy drops 22.7%","Adding a monitor role yields 85.2% on complex QA","Planner-monitor-workers design hits 85.2% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The monitor's error classification depends on a predefined error tree built from empirical data that the paper does not describe or validate; if a real error falls outside the tree or is misclassified, the monitor cannot fix it and the claimed advantage over systems without a monitor disappears.","fun_headline_variants_meta":{"raw":{"variants":["Monitor role is key to 85.2% multi-agent accuracy","Without a monitor, agent accuracy drops 22.7%","Adding a monitor role yields 85.2% on complex QA","Planner-monitor-workers design hits 85.2% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3354,"prompt_tokens":902,"completion_tokens":2452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2378}},"tokens_in":518,"tokens_out":2452,"duration_ms":18266,"temperature":1.0,"reasoning_tokens":2378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:02:25.439865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of failure logs from the two benchmarks, have independent raters label each error's branch, and compare their labels with the monitor's classifications; if agreement is low, the error tree is not doing the claimed work. A sharper test is to inject a failure type outside the paper's two branches, such as a tool-permission or prompt-security error, and observe whether the monitor can still produce a repair.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HotpotQA multi-hop reasoning benchmark used for general evaluation."},{"cited_title":"Lang C hain, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the general-purpose agent framework baseline in the code-volume comparison."}],"review_version":1}