{"id":"a8b12f9b-9d17-4d73-8d2f-dbf0228481f3","arxiv_id":"2602.00996","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DeALOG lets five specialized LLM agents cooperate through a shared text log, and the paper claims this gives competitive zero-shot accuracy on six table/text/image QA benchmarks.","lead":"A new question-answering design lets five specialized AI agents write findings to a shared text log and check each other, instead of following one central plan. The paper says this makes reasoning more robust and auditable, and it is a plausible architecture for future multi-agent systems — but its benchmark evidence is internally inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central accuracy claim rests on baseline tables whose numbers are mechanically identical across datasets/backbones and whose citations point to unrelated systems; if these rows are not genuine measurements, the 'highest or near-highest' claim collapses.","rationale":"The paper's central claim has two pillars: accuracy and robustness. Both rest on empirical comparisons that are not reproducible as presented. The reader's weakest assumption—baseline integrity—is the same one I identify. I considered whether the robustness evidence (Figure 4, Table 8) could be the single most load-bearing issue: Table 8 reports catch rates of 9–31% and repair rate 0.00 yet final EM drops only 0.03–0.07, which is internally suspicious. However, the headline claim of 'highest or near-highest exact match accuracy' and the paper's own contributions list put Tables 2–3 at the center. If those numbers are not genuine measurements, the paper fails regardless of the framework's conceptual merits. The identical-value pattern and wrong citations make the data-integrity concern concrete, not a matter of disagreeing with consensus. I am not claiming fraud; I am saying the evidence does not support the claim as documented. The concrete test—independent recomputation from raw predictions—would settle it. If the numbers check out, a CONDITIONAL accept might be defensible; absent that, REJECT stands.","tokens_in":17124,"tokens_out":5324,"duration_ms":57777,"concrete_test":"Request the per-question raw predictions and evaluation scripts for the Lumos row (or any baseline) on FinQA and FeTaQA, as promised by the reproducibility statement. Recompute exact match with the official scorer for seed 2024 under the stated BM25+miniLM retriever. If the recomputed values differ from 73/74/73 by more than rounding, or the raw predictions do not exist, Tables 2–3 are not genuine measurements.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that Tables 2 and 3 report genuine, backbone-matched measurements. This is the least secure part of the paper. Nearly every baseline row is identical across FeTaQA, FinQA, TAT-QA, MMQA, and WikiTQ (e.g., Lumos and HUSKY are 73/74/73 in every column; REWoO, Chameleon, FireAct, AutoTQA, Dater, ReAcTable, Codex, TableCritic, Planner, TiDE also show repeated triplets). Such mechanical repetition is not a property of real evaluations, where dataset difficulty and backbone interactions vary. Several baselines are not the systems the paper claims: the Lumos citation (Nigam 2025) is a user-behavior model; TiDE (Das et al. 2024) is a time-series forecaster; 'Planner' cites a paragraph-generation paper; Chameleon cites a foundation-model paper, not the reasoning framework. The promised bootstrap CIs and five-seed means are absent. There is also a direct internal contradiction: CRT-QA accuracy is 0.50 in Table 5 but 0.70 in Table 6 and the text. If these tables are schematic rather than measured, the central comparison—and with it the claim of out-performance under matched capacity—collapses, and no re-implementation of the DeALOG loop would repair it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DeALOG, a decentralized multi-agent framework for multimodal and tabular question answering. Five specialized agents (Table, Context, Visual, Summarizing, Verification) coordinate by reading and appending to a shared natural-language log, with a lightweight scheduler instead of a central planner. The authors claim that this log-mediated, verification-augmented design improves accuracy, robustness to injected noise, long-horizon reasoning, and faithfulness, and they report competitive or state-of-the-art zero-shot results on FeTaQA, FinQA, TAT-QA, WikiTQ, MMQA, and CRT-QA under three backbone LLMs. The paper also includes efficiency analyses, ablations, error taxonomies, and full implementation details and prompts in the appendix.","tokens_in":17473,"tokens_out":3590,"duration_ms":42195,"significance":"If the empirical claims were credible, the paper would make a useful contribution: it proposes a concrete, reproducible architecture for decentralized multi-agent reasoning with an auditable shared log, and it explicitly targets error propagation and verification, which are recognized weaknesses of planner-based agent pipelines. The framework design is coherent, the log schema and controller loop are described in sufficient detail to reimplement, and the fault-injection and long-horizon experiments are a reasonable way to probe robustness. However, the central claim of 'highest or near-highest exact match accuracy' rests on baseline tables whose numbers are internally inconsistent and not credible as measured results. The significance of the paper therefore depends on whether those tables can be replaced with genuine, backbone-matched measurements; in the current form, the contribution is not established.","major_comments":[{"comment":"The baseline rows in Tables 2 and 3 are mechanically identical across unrelated datasets and across all three backbone models. For example, Lumos and HUSKY are 73/74/73 in every column; TableCritic is 73/74/76 everywhere; TiDE shows 75/76/79 or 76/76/76. This pattern is not a property of real evaluations, where dataset difficulty and backbone interactions differ. The paper further states in Section 3 (Evaluation Approach) that results are means over five seeds with 95% bootstrap confidence intervals, but no confidence intervals or per-seed variation are reported anywhere. Since the paper's central claim of outperformance under matched model capacity depends entirely on these numbers, the tables cannot support the conclusion without raw measurements or error bars. This is a load-bearing issue, not a presentation detail.","section":"Tables 2 and 3, Section 3.1"},{"comment":"Several cited baselines are not the systems the names would imply. 'Lumos' is cited to Nigam (2025), a user-behavior prediction model, not a reasoning agent. 'TiDE' is cited to Das et al. (2024), a time-series forecasting paper. 'Planner' is cited to Zhang et al. (2024b), a paragraph-generation paper. 'Chameleon' is cited to 'Chameleon Team (2025)', a mixed-modal foundation-model paper, rather than the named reasoning framework. These mismatches reinforce the concern that the baseline rows in Tables 2 and 3 are schematic rather than measurements of the actual systems. The authors should provide exact source papers for each baseline and evidence that the listed scores were obtained by running those systems with the specified backbones on the same test sets.","section":"References for baselines, Section 3 (Baseline Methods) and Reference list"},{"comment":"There is a direct internal contradiction in the reported CRT-QA results. Table 5 reports CRT-QA accuracy as 0.50 (Base) and 0.50 (Gated), while Table 6 reports CRT-QA accuracy as 0.70 with a note 'Up to 10 rounds'. The text in Section 3.1 also describes CRT-QA accuracy 'dropped sharply to around 70%' for logs exceeding 8 entries. A 0.50 vs. 0.70 discrepancy is far too large to be a rounding error and is not explained. This inconsistency undermines confidence in the reliability of the reported numbers and makes it impossible to assess the paper's robustness claims.","section":"Table 5 vs. Table 6 and Section 3.1 (Impact of longer logs)"},{"comment":"The faithfulness evaluation is circular in two ways. The 'LLM Judge Support' uses an LLM judge from the same model family (LLaMA-3/Mistral/Qwen) that generated the answers, and the 'Log-Groundedness' metric is computed from the system's own append-only log, which the agents themselves produced. Without an external judge, human evaluation, or at least a different model family for adjudication, these metrics cannot validate that the answers are actually faithful to external sources. The paper's claim that 'DeALOG's faithfulness stems from multi-agent consensus' is not supported by this internal self-consistency check. This is a load-bearing point for the interpretability contribution.","section":"Table 7 and Section 3 (Role of Verification Agent)"}],"minor_comments":[{"comment":"Typo: 'We empirically validates' should be 'We empirically validate'. Also 'DeALOG, provides' in the abstract has an unnecessary comma.","section":"Abstract and Section 2"},{"comment":"The FireAct row is duplicated (3.2 and 3.6 calls both appear with different latencies). This is confusing and should be corrected.","section":"Table 4"},{"comment":"Dataset names are inconsistent: 'FeTAQA' vs. 'FeTaQA', 'WIKITQ' vs. 'WikiTQ', 'TATQA' vs. 'TAT-QA', 'Multi-ModalQA' vs. 'MultiModalQA'. Please standardize.","section":"Tables 1–3 and throughout"},{"comment":"The top part of Figure 4 is described as showing 'catastrophic error rates' but the axis labels are not visible in the text; the figure should be legible and self-contained. The bottom table uses EM, which is a different metric from catastrophic error; clarify the relationship.","section":"Section 3.1, Figure 4"},{"comment":"In the Python pseudocode, 're_engaged = True' appears after a 'continue', so it is unreachable in the shown control flow. This is a minor implementation illustration issue, but it should be fixed for clarity.","section":"Appendix A.4"}],"recommendation":"reject","confidential_remarks":"The paper's architecture is interesting, but the central empirical comparison appears to be based on schematic or non-genuine numbers: baseline accuracy values are identical across unrelated datasets and backbones, several baseline citations point to unrelated papers, and there is a direct 0.50 vs. 0.70 contradiction for CRT-QA. These issues are load-bearing for the paper's core claim and cannot be fixed by minor revision; they require re-running the baselines and reporting genuine measurements. I would recommend rejection, though I would be open to reconsidering a substantially revised submission with credible, reproducible experiment tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that the framework description is the strongest part and the numbers are the problem. The shared-log multi-agent design is a reasonable adaptation of blackboard systems: five specialized agents, a lightweight scheduler, typed log entries, provenance, truncation, and a verification pass. If you want to build a planner-free log-mediated QA system, the pseudocode and agent prompts give you a clear blueprint. The ablations (partial log visibility hurts, verification helps) are plausible and the limitations section is honest about latency and long-log degradation.\n\nBut the empirical backbone is not believable. Tables 2 and 3 show nearly every baseline with identical accuracy values across FeTaQA, FinQA, TAT-QA, MMQA, and WikiTQ, and across three backbones. Lumos and HUSKY are 73/74/73 in every column. That is not what real benchmark measurements look like. Dataset difficulty and backbone interactions vary. This mechanical pattern alone is enough to undermine the claimed comparisons.\n\nIt gets worse. Several baseline citations point to unrelated systems: the Lumos reference is a user-behavior model, TiDE is a time-series forecaster, Planner is a paragraph-generation paper, and the Chameleon citation is a foundation-model paper, not the reasoning framework. The CRT-QA accuracy is listed as 0.50 in Table 5 and 0.70 in Table 6 with no explanation. The paper promises 95% bootstrap CIs and five-seed means, but no CIs appear anywhere. No code or data artifact is released despite the claim of an anonymous codebase.\n\nThese are not cosmetic issues. The central claim of competitive or state-of-the-art accuracy rests on those tables, and those tables cannot be trusted. The architecture might still be worth studying in isolation, but as submitted the paper does not provide reliable evidence for its own contribution.\n\nMy recommendation: desk reject. If the authors rerun the baselines properly with correct citations, release code, and fix the internal contradictions, this could become a solid engineering paper. In its current form, engaging with it would reward carelessness.","headline":"The framework write-up is detailed and the architecture is plausible, but the baseline tables look mechanically generated and several citations point to unrelated papers, so the central accuracy claim does not survive scrutiny.","tokens_in":18014,"tokens_out":2693,"would_cite":false,"duration_ms":32226,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In DeALOG, specialized LLM agents with no central planner coordinate by reading and appending to a shared natural-language log; the paper argues that this decentralized design exposes intermediate reasoning, enables peer verification, and i","keywords":["multi-agent reasoning","question answering","shared memory","natural-language log","multimodal QA","verification","decentralized systems","blackboard architecture"],"falsifier":"Re-run the stated experiments on the stated datasets and backbones (e.g., FeTaQA, FinQA, MMQA, WikiTQ with LLaMA-3 8B, Mistral 7B, Qwen-3 8B) and compare DeALOG against the listed baselines under the same retriever and prompts. The robustness claim could be settled by injecting 10–30% corruption into log entries and checking whether DeALOG's exact match remains near 0.73 while planner baselines drop to 0.55–0.66; the data integrity of Tables 2–3 can be checked by verifying whether baseline accuracies vary across datasets and backbones as real measurements would.","tokens_in":16970,"feed_emoji":"📋","tokens_out":6740,"duration_ms":66243,"temperature":0.7,"pith_summary":"The paper introduces DeALOG, a decentralized multi-agent framework for question answering over tables, text, and images. Instead of a central planner that decomposes the question and assigns steps, five specialized agents — Table, Context, Visual, Summarizing, and Verification — read and append to a shared, append-only natural-language log that serves as persistent memory. The authors argue that this log-mediated design exposes intermediate reasoning, lets peers catch each other's errors, and avoids the error amplification that plagues planner pipelines. They report competitive or best exact-match accuracy on FeTaQA, FinQA, MMQA, and WikiTQ across three backbones, with explicit verification, global log visibility, and agent specialization each contributing to the gains, and they acknowledge weaker results on TAT-QA and higher latency. A sympathetic reader takes away a concrete architectural claim: centralized planning is not necessary for strong, robust multi-hop QA if agents share a transparent evidence log.","feed_headline":"Drop the central planner: shared-log agents match top QA baselines","feed_subtitle":"Decentralized agents catch each other's errors; accuracy holds on corrupted and long-chain questions.","key_machinery":"The shared log: an append-only list of typed entries — LOOKUP, QUOTE, VISUAL, SUMMARY, ANSWER, FLAG, OK — each tagged with the producing agent and provenance metadata (table row/column, document span, image ID). All agents see the whole log; a lightweight scheduler (not a planner) controls turn-taking, duplication filtering, and the stop condition, and the VerificationAgent performs consistency checks over the log, triggering at most one re-engagement round when it flags an error. The log is what lets coordination emerge without a central plan and what makes the reasoning auditable.","core_discovery":"The central claim is that replacing centralized planning with log-mediated collaboration improves robustness and interpretability without sacrificing accuracy. In the proposed system, each agent independently decides when to act based on the current log; the SummarizingAgent produces a candidate answer, and the VerificationAgent recomputes calculations, checks units, and confirms the answer is supported by log entries, flagging inconsistencies for one corrective re-engagement round. Under matched model capacity, the paper reports DeALOG achieving the highest or near-highest exact match on FeTaQA, FinQA, MMQA, and WikiTQ, and maintaining higher exact match than re-planning Planner and Plan→Lo","pith_inferences":["If the robustness results reproduce, a testable extension is that log-mediated systems should outperform planner systems increasingly as task depth grows, because errors are localized to individual log entries rather than propagated through a fixed plan.","The shared log is itself an audit trail, so the architecture could be adapted to QA settings where answers must be justified to a non-expert; the paper measures log-groundedness but does not exploit the log for user-facing explanations.","The low detection rate for visual/OCR errors suggests a concrete improvement the paper leaves open: a verification step that revisits the raw image or calls a visual question-answering model instead of trusting the OCR/caption text.","Because all agents write and read the same log, DeALOG could also serve as a substrate for heterogeneous agents with different backbones or tools, each contributing typed entries; the paper notes tool integration is possible but does not demonstrate it."],"forward_implications":["If DeALOG's claims hold, planner-free log-mediated coordination is a viable alternative to planner–executor architectures for multimodal QA, with comparable or better accuracy under matched backbone size.","Robustness to error propagation improves: exact match stays at 0.73 under 30% corruption of log entries, versus 0.55 for the re-planning Planner and 0.66 for a Plan→Log hybrid.","Long-horizon reasoning degrades more gracefully: 0.77 EM on chains of 8+ operators, where the planner baselines fall to 0.50 and 0.64.","Explicit verification is a major contributor: arithmetic and unit errors are caught at 88%, and the one-shot re-engagement corrects retrieval omissions; visual/OCR errors remain a weak point at 17.1% detection.","Learned gating can cut agent turns and tokens by up to 18% without meaningful accuracy loss, partially offsetting the latency cost of sequential agent calls."],"fun_headline_variants":["Shared-log agents match planner baselines without central control","No central planner: log-mediated agents keep QA accuracy","Decentralized multi-agent log beats re-planning on multimodal QA","Agent log instead of controller: robust multimodal QA","Shared natural-language log: agents collaborate, verify, match baselines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the reported baseline accuracy numbers are genuine, backbone-matched measurements; in Tables 2 and 3 many baseline rows show identical values across all datasets and all three backbones, and if those numbers are schematic rather than measured, the claim of out-performance is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["Shared-log agents match planner baselines without central control","No central planner: log-mediated agents keep QA accuracy","Decentralized multi-agent log beats re-planning on multimodal QA","Agent log instead of controller: robust multimodal QA","Shared natural-language log: agents collaborate, verify, match baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":947,"prompt_tokens":644,"completion_tokens":303,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":388,"completion_tokens_details":{"reasoning_tokens":222}},"tokens_in":388,"tokens_out":303,"duration_ms":3847,"temperature":1.0,"reasoning_tokens":222,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:48:15.799519+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the stated experiments on the stated datasets and backbones (e.g., FeTaQA, FinQA, MMQA, WikiTQ with LLaMA-3 8B, Mistral 7B, Qwen-3 8B) and compare DeALOG against the listed baselines under the same retriever and prompts. The robustness claim could be settled by injecting 10–30% corruption into log entries and checking whether DeALOG's exact match remains near 0.73 while planner baselines drop to 0.55–0.66; the data integrity of Tables 2–3 can be checked by verifying whether baseline accuracies vary across datasets and backbones as real measurements would.","supporting_citations":[],"review_version":1}