{"id":"bebd5a96-c590-44d6-b8e2-2bdbcff869bf","arxiv_id":"2608.11965","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A mixed-method experience report on LLM-based multi-agent frameworks for software engineering: broad feature coverage, weak monitoring support, and no clear quality winner on a README-summarization task, with incomplete quantitative evidence.","lead":"Software engineers building teams of AI agents can choose from many open-source frameworks. This paper compares eight of them and reports that most cover basic agent features but lack monitoring, and that on a README-summarization task the frameworks differed little in output quality, though that empirical result is not fully supported by the paper's own data.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'no significant difference in ROUGE' is contradicted by the paper's own pairwise Wilcoxon results and the missing Metagente baseline.","rationale":"The reader's weakest assumption was that the framework reimplementations are faithful, behaviorally equivalent ports, so the standardized settings and minor prompt modifications do not differentially affect ROUGE scores. That is a plausible secondary concern. However, the more load-bearing problem is more direct: the paper's own Figure 5 reports many significant pairwise p-values, so the abstract claim of 'no significant difference' is internally contradicted no matter how faithful the ports are. The reader's rationale also identifies this via the missing Metagente baseline and the four-versus-five framework inconsistency, which is why I mark agreement as partial rather than full. The qualitative contribution—the feature matrix, installation/orchestration assessment, and lessons learned—is informative and could survive a revision. But the abstract and central claim hinge on the quantitative ROUGE comparison, and that comparison is not reported consistently: the baseline named in the methodology never appears, the framework count changes between sections, and the efficiency results promised in the abstract are missing from the body. These are mechanical, verifiable inconsistencies rather than matters of interpretive disagreement. A revision that reports the complete quantitative results, including the Metagente baseline and efficiency measures, or that softens the abstract to describe a qualitative comparison with a limited quantitative illustration, would be necessary before the central claim as stated can be accepted.","tokens_in":31108,"tokens_out":3759,"duration_ms":36908,"concrete_test":"Using the replication package (De Oliveira et al. 2025), recompute the ROUGE scores for TS10 and TS50 for AutoGen, AutoGPT, Dify, and Semantic Kernel, then run a Friedman omnibus test across the four frameworks followed by pairwise Wilcoxon comparisons that include the Metagente baseline. If the omnibus test rejects at p<0.05, or if any pairwise p<0.05 survives when the baseline is included, the abstract's no-significant-difference claim fails. The check requires only the published code, data, and the Metagente outputs; if those baseline outputs are not in the package, that absence itself confirms that the central comparison was not actually reported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim, repeated in the abstract and conclusion, is that 'there is no significant difference in terms of ROUGE scores' across frameworks. Section 4.4.1's Figure 5 directly contradicts this: many pairwise Wilcoxon p-values are below 0.05, e.g., for TS10 ROUGE-1 AutoGen vs AutoGPT p=0.001, AutoGen vs Dify p=0.001, AutoGPT vs Dify p=0.001, and AutoGen vs Semantic Kernel p=0.014; similar significant differences appear across ROUGE-2, ROUGE-L, and TS50. The paper's own data therefore show that some frameworks differ significantly on the reported metric. Additionally, Section 4.3 states that the Wilcoxon test compares 'each MAS framework with the Metagente framework,' but Figure 5 contains only the five reimplementations and no Metagente row or column; the promised baseline comparison is absent. With many pairwise differences and no baseline, the practitioner-facing conclusion that framework choice does not affect summary quality is unsupported. This is an internal inconsistency in the reported evidence, not a disagreement with external consensus. The qualitative feature matrix and lessons learned may still be useful, but the quantitative claim that anchors the abstract needs the reported statistics to support it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is an experience report on LLM-based multi-agent system (MAS) frameworks for software engineering. It combines a qualitative analysis of open-source frameworks (feature and characteristic matrices, thematic analysis) with a quantitative evaluation in which the authors re-implement their previously proposed Metagente README-summarization pipeline in several frameworks (AutoGen, AutoGPT, Dify, Semantic Kernel, plus Semantic Kernel Chat) and compare ROUGE-1/2/L scores on training sets TS10 and TS50 and a test set ES. The abstract and conclusion state that there is no significant difference in ROUGE scores across frameworks, while execution time varies significantly. The paper also provides lessons learned and challenges for practitioners, and a replication package is referenced.","tokens_in":31260,"tokens_out":7291,"duration_ms":63390,"significance":"If the central claim were supported, the paper would be a useful practical guide: practitioners could choose a MAS framework based on qualitative features, documentation, and orchestration fit rather than expected summary quality, because README-summarization effectiveness would be indistinguishable across frameworks. The qualitative framework matrices and lessons learned are valuable resources, and the authors follow a transparent process (contamination audit, normality testing, Wilcoxon tests). However, the quantitative contribution is internally inconsistent: the abstract's 'no significant difference' claim is contradicted by the many significant pairwise p-values in Figure 5, the Metagente baseline promised in Section 4.3 is absent from the reported results, and the claimed efficiency finding is not reported anywhere. These load-bearing issues undermine the paper's main empirical conclusion.","major_comments":[{"comment":"The abstract and conclusion state that 'there is no significant difference in terms of ROUGE scores' across frameworks, but Figure 5 reports many significant pairwise Wilcoxon p-values, e.g., TS10 ROUGE-1 AutoGen vs AutoGPT p=0.001, AutoGen vs Dify p=0.001, AutoGPT vs Dify p=0.001, and AutoGen vs Semantic Kernel p=0.014, with similar significant results in Figures 5(b)-(f). The claim is therefore contradicted by the paper's own reported statistics. Either the significance analysis is incorrect or the conclusion must be substantially qualified (e.g., 'no framework is consistently and significantly better on all metrics').","section":"Abstract and §4.4.1 (Figure 5)"},{"comment":"Section 4.3 says the Wilcoxon signed-rank test is used 'to compare the performance of each MAS framework with the Metagente framework,' but Figure 5 and Table 19 contain only the five reimplementations; no Metagente row or column appears. The promised baseline comparison is missing, so the quantitative evaluation cannot be checked against the reference system and the 'no significant difference' claim has no baseline anchor.","section":"§4.3 vs Figure 5/Table 19"},{"comment":"The abstract claims that 'time taken to complete the task varies significantly across the frameworks,' but no section presents the efficiency data (token usage, number of requests, usage time). Section 4.4 presents only Effectiveness, and the cross-reference to 'Section??' for efficiency is unresolved. A claimed finding of the paper is therefore unreported.","section":"§4.4 and abstract"},{"comment":"The feature classification in Table 6 is attributed to 'a single author' (Section 3.2) with the classification distributed among authors; no inter-rater reliability measure is reported for the qualitative matrices. Additionally, §3.3 (Addressing RQ3) says the quantitative evaluation selected 'four tools,' but Table 19 and Figure 5 include five implementations (Semantic Kernel Chat is added). The paper should explain why there is a fifth implementation and how the selection criteria apply.","section":"§3.2 and §3.3"},{"comment":"The empirical evaluation re-implements the authors' own Metagente pipeline in each framework using 'minor modifications' to prompts, standardized models, 15 iterations, and a ROUGE-L threshold of 0.7 adopted from the original study. The comparison is therefore between the authors' ports of a single pipeline, not between idiomatic uses of the frameworks. The conclusion that framework choice does not affect summary quality should be scoped to this 'Metagente-style port' context, and the circularity risk—the benchmark infrastructure comes entirely from the authors' prior work—should be discussed explicitly.","section":"§4.1.1"}],"minor_comments":[{"comment":"The manuscript contains unresolved 'Section ??' cross-references for the lessons-learned section and the efficiency section; these must be fixed.","section":"§1, §4.3, §4.4"},{"comment":"The text first says 'boxplots in the right side of Figure 4' and then refers to 'violin plots'; Figure 4 should be described consistently.","section":"§4.4.1"},{"comment":"Table 6 uses unclear symbols (e.g., 'G #', '#') that are not defined; the legend should use unambiguous glyphs (e.g., full, partial, not supported) consistently.","section":"§3.2, Table 6"},{"comment":"The search window is stated as January 2023 to January 2025, but the reference list includes works dated 2026 (e.g., Wang and Yin 2026; Zeshan et al. 2026); clarify the search and update the reported window.","section":"§3.1"},{"comment":"In the conclusion, 'recommendation accuracy' appears to be a typo for 'summarization accuracy' or similar; revise for clarity.","section":"§5"},{"comment":"Table 12 lists 'PT' for AutoGPT's real-time logs without defining the value; use a defined symbol or spell out the partial support.","section":"§3.3.6, Table 12"},{"comment":"The bullet list containing 'Impacts', 'Selection of frameworks', 'Limitations', and 'Future Extension' appears within §4.4.1 after the TS10 results; this material should be moved to a dedicated discussion section to avoid confusing results with discussion.","section":"§4.4.1"}],"recommendation":"reject","confidential_remarks":"The internal inconsistencies are verifiable from the paper's own figures and tables, so they are not a matter of reviewer preference. The authors' reliance on their own Metagente infrastructure as the benchmark is a further concern for independence. If the authors can provide the missing Metagente baseline and the efficiency analysis, a revised version might be reconsidered, but the current manuscript's central quantitative claim is not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper has a genuinely useful qualitative comparison of 16 LLM-based MAS frameworks from a developer's perspective, and the lessons learned (monitoring is immature, low-code frameworks are still limited) are worth having. The feature matrix and the developer-facing characteristics are a legitimate contribution, and the replication package is a plus. That part deserves credit.\n\nThe quantitative part, however, does not support its own headline. The abstract and conclusion say there is no significant ROUGE difference across frameworks, but Figure 5 reports many significant pairwise Wilcoxon p-values (e.g., AutoGen vs. AutoGPT p=0.001 for TS10 ROUGE-1). Section 4.3 says the test compares each framework to the Metagente baseline, yet no Metagente row or column appears in the results. The efficiency results promised in the abstract (\"time taken ... varies significantly\") are missing, with a dangling section reference. The framework count shifts from four to five, and a 'Semantic Kernel Chat' variant appears only in the results. These are not stylistic quibbles; they break the central empirical claim as stated. The reimplementation also reuses the authors' own Metagente pipeline, prompts, datasets, and thresholds, so the ports need to be demonstrably faithful for the comparison to be interpretable.\n\nStill, I wouldn't kill it. The qualitative matrix and lessons learned stand on their own, and the problems are fixable: either report the full quantitative results, including the baseline and efficiency data, or rewrite the abstract to position the quantitative part as an illustrative comparison rather than a null result. As it stands, the empirical conclusion is unsupported.\n\nI'd send this to peer review with major revision. It deserves referee time because the qualitative comparison fills a real gap, and the authors are positioned to fix the empirical section. But I would not accept it as is.","headline":"A useful qualitative framework comparison whose quantitative headline is contradicted by its own significance tests.","tokens_in":31884,"tokens_out":3517,"would_cite":false,"duration_ms":33640,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An experience report argues that the choice of LLM-based multi-agent framework barely affects task output quality, because five reimplementations of the same pipeline yield statistically equal ROUGE scores.","keywords":["multi-agent systems","large language models","software engineering","framework comparison","README summarization","ROUGE","mixed-methods study","experience report"],"falsifier":"Replace the per-framework adapted prompts with one identical, carefully worded prompt set and rerun the optimization pipeline on the same datasets across all frameworks; if significant pairwise ROUGE differences appear under identical prompts, the paper's no-difference conclusion is an artifact of prompt adaptation rather than a fact about the frameworks. A simpler check is to inspect the published per-framework prompts: if they differ in output-format constraints, few-shot examples, or task instructions, the standardization premise is already violated.","tokens_in":30859,"feed_emoji":"🤖","tokens_out":9298,"duration_ms":80819,"temperature":0.7,"pith_summary":"Through a mixed-method study, the paper tries to establish that the choice among popular open-source frameworks for building LLM-based multi-agent systems has little effect on the quality of the system's output. The authors qualitatively analyze sixteen frameworks against a ten-feature taxonomy of multi-agent capabilities, then reimplement the same README summarization pipeline on four frameworks (plus a chat-based variant) and find the resulting ROUGE scores statistically indistinguishable. If this is right, practitioners can choose a framework based on documentation, orchestration style, and monitoring support rather than expected output quality. The study also finds that advanced capabilities such as agent telemetry and benchmarking remain largely missing, while execution time varies significantly across frameworks.","feed_headline":"LLM agent frameworks produce equal summary quality","feed_subtitle":"ROUGE scores are statistically equal across frameworks, so docs and orchestration fit decide the choice.","key_machinery":"The load-bearing machinery is the standardized reimplementation of a reference multi-agent pipeline, called Metagente in the paper, across the frameworks under comparison. The pipeline assigns four agent roles—an Extractor, a Summarizer, a Teacher, and a Prompt Creator—that run an iterative prompt-optimization loop capped at 15 iterations with a success threshold of 0.7 on ROUGE-L, followed by a two-agent evaluation stage. By keeping the same LLM models, temperatures, and dataset across all ports, the study isolates the framework as the experimental variable; the Wilcoxon signed-rank test, a non-parametric test for paired samples, then converts the ROUGE score distributions into pairwise significance judgments. Qualitatively, the analysis is carried by a ten-feature taxonomy of multi-agent capabilities—core architecture, role specification, tool support, monitoring, human feedback, and related dimensions—applied to each framework's documentation and feature set.","core_discovery":"On the paper's own terms, the central discovery is that the frameworks' effectiveness on a common multi-agent task is statistically indistinguishable. The authors reimplemented the same prompt-optimization pipeline for GitHub README summarization on four open-source frameworks (and a fifth chat-based variant), holding the underlying LLMs, temperatures, iteration limits, and the ROUGE-L threshold fixed. Pairwise Wilcoxon signed-rank tests over ROUGE-1, ROUGE-2, and ROUGE-L distributions mostly fail to reach significance, with two of the frameworks behaving comparably. The paper couples this with a qualitative finding that the frameworks cover foundational multi-agent capabilities well—role specification, orchestration, message handling—while advanced features such as telemetry, monitoring, and benchmarking remain missing.","pith_inferences":["If the quality-indifference result generalizes beyond summarization, the practical bottleneck in building LLM-based multi-agent systems is likely the prompt design and orchestration logic, not the framework, which would shift developer attention toward composition patterns and away from framework selection.","A natural extension is a controlled ablation in which the same agent logic is ported by different development teams; robustness of the no-difference result across implementers would strengthen the interchangeability claim, while sensitivity would caution against over-generalizing.","ROUGE scores on short summaries may be too coarse to reveal framework differences; pairing ROUGE with semantic similarity metrics or human judgments could confirm the equivalence or expose differences that n-gram overlap misses."],"forward_implications":["Practitioners can select a multi-agent framework on documentation quality, low-code versus high-code fit, and orchestration style, because output quality on summarization-type tasks does not differentiate the frameworks.","Teams building production systems should budget for external telemetry and monitoring, since the study finds advanced observability features largely missing across the selected frameworks.","Execution-time variation means framework choice still affects rapid-prototyping speed and running cost, even when output quality is equal.","The ten-feature taxonomy provides a reusable checklist for evaluating future multi-agent frameworks as the tooling landscape evolves.","The no-difference result invites replication on other software-engineering tasks, such as code generation or test generation, to see whether it holds beyond summarization."],"supporting_citations":[{"why":"Defines the Metagente architecture and the standardized settings (15 iterations, ROUGE-L threshold 0.7, model choices) that all five reimplementations port, making it the direct baseline for the quantitative comparison.","marker":"(Nguyen et al. 2025)"},{"why":"Supplies the original README summarization dataset that, after cosine-similarity filtering, yields the training and evaluation sets used in the experiment.","marker":"(Doan et al. 2023)"},{"why":"Defines the ROUGE-1, ROUGE-2, and ROUGE-L metrics used to measure summary effectiveness.","marker":"(Lin 2004)"},{"why":"Provides the normality test the authors use to justify selecting the non-parametric Wilcoxon signed-rank test for the pairwise comparisons.","marker":"(Shapiro and Wilk 1965)"},{"why":"Is the signed-rank test used for pairwise framework comparisons and is the statistical basis for the 'no significant difference' conclusion.","marker":"(Wilcoxon 1945)"},{"why":"Contributes the foundational multi-agent concepts (roles, orchestration, memory, tools) that the qualitative ten-feature taxonomy operationalizes.","marker":"(He et al. 2025)"}],"fun_headline_variants":["LLM agent frameworks tie on ROUGE scores","Agent framework choice has no effect on summaries","Equal ROUGE scores, missing telemetry in agent frameworks","ROUGE parity across LLM agent frameworks, telemetry lacking","Agent frameworks produce statistically equal summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quantitative comparison assumes that the five reimplementations are faithful, behaviorally equivalent ports of the same multi-agent logic, so the 'minor modifications' to prompts and the varying levels of framework support do not differentially affect the measured ROUGE scores; if the ports differ in implementation quality, the no-significant-difference result could be an artifact of the reimplementation rather than a property of the frameworks.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent frameworks tie on ROUGE scores","Agent framework choice has no effect on summaries","Equal ROUGE scores, missing telemetry in agent frameworks","ROUGE parity across LLM agent frameworks, telemetry lacking","Agent frameworks produce statistically equal summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000554,"raw_usage":{"total_tokens":2646,"prompt_tokens":957,"completion_tokens":1689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1614}},"tokens_in":573,"tokens_out":1689,"duration_ms":10977,"temperature":1.0,"reasoning_tokens":1614,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:20:43.613348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the per-framework adapted prompts with one identical, carefully worded prompt set and rerun the optimization pipeline on the same datasets across all frameworks; if significant pairwise ROUGE differences appear under identical prompts, the paper's no-difference conclusion is an artifact of prompt adaptation rather than a fact about the frameworks. A simpler check is to inspect the published per-framework prompts: if they differ in output-format constraints, few-shot examples, or task instructions, the standardization premise is already violated.","supporting_citations":[],"review_version":1}