{"id":"760966e3-59b9-4620-b947-7b56b6903bd0","arxiv_id":"2505.14848","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Assigning separate MQM error categories to different AI agents, then merging their annotations with an editor agent, improved machine translation scores over zero-shot and single-agent baselines, with model-dependent human preference.","lead":"MAATS is a translation pipeline where separate AI agents each check one kind of error, accuracy, fluency, style, terminology, and an editor agent combines their findings into a corrected translation. Tested on six languages and three AI models, it beat plain single-pass translation on most automatic scores, but human rankings only clearly favored it for one of the three models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MAATS's improvement over its single-agent baseline is confounded by a roughly 7x difference in LLM calls; without a compute-matched single-agent baseline, the gain cannot be attributed to MQM decomposition, and the human-eval claim overstates sub-50% win rates.","rationale":"The reader identified compute mismatch as the weakest assumption, and I agree; it is the load-bearing concern because it directly blocks the central causal claim that MQM decomposition, rather than additional inference, produces better translations. My read of the full text confirms the single-agent baseline uses one refinement pass while MAATS uses seven evaluators plus an editor; the paper's Appendix A even calls the baseline 'essentially an ablated version', but it is not compute-ablated. I also note that the human-evaluation results in Fig. 5 do not support the abstract's wording: two of three models show sub-50% win rates against single-agent, and no human significance test is reported. This is an overstatement, but the central architectural claim is still testable. Given the repository, exact test subsets, and human reference data are not verifiable from the manuscript, the CONDITIONAL verdict remains appropriate: acceptance should require a compute-matched baseline and corrected human-evaluation reporting. The paper deserves credit for a clear design, detailed prompts, 11 translation directions, multiple metrics, and honest qualitative case studies; the concern is about attribution, not fraud or sloppy execution.","tokens_in":15384,"tokens_out":3661,"duration_ms":34189,"concrete_test":"Run a compute-matched baseline on the same 11 directions: for each sentence, give the same base LLM the single-agent review prompt for N=8 sequential self-refinement rounds (or 7 independent critique passes followed by an editor), matching MAATS's ~9 calls and comparable token budget. Compare COMET/BLEURT and human ranking (EN->ZH) with paired bootstrap and confidence intervals. If the compute-matched single-agent matches MAATS's gains over the one-pass baseline, the MQM multi-agent architecture is not the cause; if MAATS still wins, the attribution concern is resolved. Also report the human win-rate test (e.g., exact binomial CI) for Fig. 5 to verify whether Claude/Gemini results differ from chance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the modular MQM-category evaluator plus editor architecture causes translation-quality gains, not just more refinement. Appendix A states the single-agent baseline performs one self-refinement pass (initial translation plus one review/refine; ~2 LLM calls per sentence), while MAATS runs a translator, seven category evaluators, and an editor (~9 calls). No compute-matched baseline is reported, so the observed COMET/BLEURT gains over single-agent could be explained by inference budget alone. The paper's assertion that the single-agent is 'essentially an ablated version' is not an ablation of compute: it removes both the MQM decomposition and 7 of 9 LLM calls simultaneously. The human-evaluation evidence is also internally inconsistent with the abstract: Fig. 5 gives MAATS vs single-agent win rates of 62.1% (GPT), 48.3% (Claude), and 49.2% (Gemini); two of three are below chance, and no significance test or confidence interval is reported. Thus the abstract's 'statistically significant gains in ... human assessments' is unsupported. The most load-bearing missing piece is a budget-matched comparison; without it, the architecture's causal role is unproven.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces MAATS, a multi-agent translation refinement system in which a translator agent produces an initial translation, seven specialized agents annotate errors using MQM categories, and an editor agent produces a final revision. The authors evaluate MAATS against zero-shot and single-agent self-refinement baselines across three LLMs and eleven English/other language directions using BLEU, METEOR, BLEURT, and COMET, a confusion-matrix comparison with human MQM annotations, a small human preference ranking for English-to-Chinese, and qualitative case studies. The central claim is that the MQM-decomposed multi-agent architecture yields consistent, statistically significant translation-quality gains over single-agent self-correction.","tokens_in":15604,"tokens_out":5557,"duration_ms":45531,"significance":"The paper is potentially useful: it releases code and data, documents all prompts in appendices, and the qualitative case studies are instructive. The MQM-decomposed evaluation idea is a sensible and interpretable way to structure LLM-based translation refinement, and the limitation section honestly acknowledges dimensions such as tone, irony, and stance that the MQM taxonomy does not cover. However, the evidence for the central causal claim is currently incomplete: the single-agent baseline is not matched in inference budget, the human preference results are below chance for two of three models, and the significance claims in the abstract exceed what Table 2 and Figure 5 show. With a compute-matched baseline and corrected statistical reporting, the architecture claim could become solid.","major_comments":[{"comment":"The single-agent baseline performs one self-refinement pass (initial translation plus one review/refine, roughly two LLM calls per sentence), while MAATS uses a translator, seven MQM evaluators, and an editor (roughly nine calls). The appendix calls the baseline 'essentially an ablated version of MAATS,' but this comparison removes both the MQM decomposition and most of the inference budget simultaneously. Since the abstract credits the gains to the 'modular agent roles' and the MQM framework, a budget-matched baseline (e.g., a single agent given an equal number of self-refinement passes or an equal token budget) is required to rule out 'more refinement helps' as the explanation for the automatic-metric improvements in Table 2 and Figure 2. This missing control is load-bearing for the title claim.","section":"Appendix A and Section 2.3"},{"comment":"The abstract states MAATS achieves 'statistically significant gains in both automatic metrics and human assessments,' but the reported human win rates against the single-agent baseline are 62.1% for GPT-4o, 48.3% for Claude-3-haiku, and 49.2% for Gemini-2.0-flash. Two of the three rates are below chance, no significance test or confidence interval is reported, and the Borda-count aggregation over three annotators does not by itself support significance. The sentence 'MAATS consistently outperformed both baselines' in the human-evaluation paragraph is therefore not supported by the data; the claim should be limited to GPT-4o or re-analyzed with per-sentence bootstrap or sign tests.","section":"Section 3, Figure 5 and Appendix C"},{"comment":"The paper's own pairwise significance table shows many language/model combinations with no significant gains on any metric (e.g., Gemini EN_DE 0/4, Claude RU_EN 1/4) and several with only BLEU significance, so the unqualified 'statistically significant gains' in the abstract is broader than the evidence. The aggregate claim should be conditioned on model, language direction, and metric, and the manuscript should report the bootstrap confidence intervals or effect sizes that underlie the summaries in Figures 10-12.","section":"Section 3, Table 2"},{"comment":"The confusion-matrix evaluation compares MAATS's error annotations with human MQM labels using the same MQM taxonomy and severity definitions that MAATS's evaluator prompts embed. This creates a partial circularity: higher true-positive and lower false-negative counts relative to the single-agent baseline may reflect that MAATS is explicitly prompted to emit that label inventory. The claim that MAATS is 'better at error detection' should be restated as 'better at reproducing MQM-format annotations,' and should be supplemented by an evaluation not built on the same taxonomy (e.g., direct human quality judgments on the same outputs) or by reporting the severity and false-positive distribution that the authors acknowledge.","section":"Section 3, Table 1 and Appendix H"}],"minor_comments":[{"comment":"The caption contains the typo 'Zeor-Shot'; it should read 'Zero-Shot.'","section":"Figure 1 caption"},{"comment":"The text uses 'ANOV A' in several places; this should be 'ANOVA.'","section":"Section 2.3 and Appendix D"},{"comment":"The caption says 'Full-scale versions see Table 16,' but the full-scale versions are shown in Figure 16; there is no Table 16.","section":"Figure 2 caption"},{"comment":"The claim of 'an increase about 450%' is arithmetically incorrect: 41,547 divided by 9,217 is approximately 4.51, which is a roughly 351% increase, not 450%.","section":"Section 3"},{"comment":"The prompt templates contain inconsistent placeholders, such as the Style prompt using '[style]' instead of severity labels, and several templates say 'If doesn't exit' instead of 'If doesn't exist.' These should be harmonized.","section":"Appendix H"},{"comment":"The manuscript alternates between 'Locale Convention' and 'Locale Conventions' and between 'Audience Appropriateness' and 'Audience Appropriate'; the terminology should be made consistent.","section":"Throughout"},{"comment":"The appendix says each annotator evaluated approximately 150 samples per LLM system; it should specify whether the three annotators evaluated the same or disjoint sentences, since this affects how Borda scores aggregate and whether inter-annotator agreement can be reported.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the compute confound is well-founded and lands on the central claim. I did not find evidence of deliberate overclaiming; the paper simply needs a budget-matched baseline and tightened statistical claims. The scope fits the journal; I would encourage the authors to run the control experiment with several self-refinement rounds and to rephrase the human-evaluation section. Note also that the abstract's 'statistically significant gains in human assessments' should be corrected even if no new human data are collected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful idea—decompose MQM critique into per-category evaluator agents and let an editor synthesize the edits—and the case studies show it catches things a single self-critical pass misses. But the central causal claim is not yet supported.\n\nThe main problem is compute. The single-agent baseline runs about 2 LLM calls per sentence (initial translation plus one self-refinement pass). MAATS runs about 9 (translator + seven evaluators + editor). That's a 4.5x gap, not the 7x the stress-test says, but it is still a serious confound. The paper argues the single-agent is 'essentially an ablated version,' but it removes both the MQM decomposition and most of the inference budget simultaneously. Without a compute-matched baseline (e.g., three or four self-refinement rounds), the automatic-metric gains could just reflect more passes.\n\nThe human evaluation is also overstated. The abstract claims 'consistently outperformed' on human assessments. Figure 5 shows MAATS beats single-agent in only 48.3% (Claude) and 49.2% (Gemini) of comparisons—below chance. Only GPT-4o shows a clear preference (62.1%). No significance test or CI is reported for these win rates.\n\nA smaller but real issue: the GPT row of the confusion matrix (Table 1) has TP+FN totals for Accuracy, Fluency, Style, and Terminology that are lower than the single-agent row, while Claude and Gemini match. That suggests either a different test subset or a data error, and it undermines the GPT confusion-matrix comparison.\n\nWhat is good: the MQM-aware prompt templates are detailed; the case studies are genuinely illustrative; the limitation section is honest about emotional and pragmatic gaps; and code/data links are provided. The architecture is plausible and worth pursuing. The flaws are fixable rather than fatal.\n\nNet: I would send this to peer review. A serious referee could push the authors to add a compute-matched baseline, fix the human-eval reporting, and clean up Table 1. The paper is not there yet, but it is a legitimate contribution to the multi-agent MT discussion.","headline":"Promising multi-agent MQM translation system, but compute-matched baselines and honest human-eval reporting are needed before the causal claims hold.","tokens_in":16187,"tokens_out":5219,"would_cite":false,"duration_ms":40392,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MAATS: seven specialized MQM agents beat one self-correcting LLM translator.","keywords":["multi-agent systems","machine translation","multidimensional quality metrics","MQM","LLM refinement","translation evaluation","COMET"],"falsifier":"Run the single-agent baseline through multiple self-refinement rounds so it uses a comparable number of LLM calls per sentence as MAATS, then compare COMET, BLEURT, and human rankings on the same test sets. If the budget-matched single agent matches or beats MAATS, the improvement cannot be credited to the MQM multi-agent design.","tokens_in":15138,"feed_emoji":"🌐","tokens_out":9711,"duration_ms":77189,"temperature":0.7,"pith_summary":"The paper introduces MAATS, a translation pipeline in which a translator agent produces a first draft, seven evaluator agents each check it against one dimension of the Multidimensional Quality Metrics (MQM) framework, and an editor agent turns their annotated findings into a revised translation. The central claim is that this division of labor improves translation quality more than asking a single model to critique and correct its own output. The authors report statistically significant gains over zero-shot and single-agent baselines on automatic metrics and in rankings by professional translators, with the largest improvements on semantic accuracy, locale adaptation, and linguistically distant language pairs. The broader point is that an interpretable error taxonomy can be used not just to measure translation quality but to organize multiple LLM agents into a human-like translator-editor workflow.","feed_headline":"Seven MQM agents beat one self-correcting LLM translator","feed_subtitle":"Specialized error-category reviewers plus an editor outperform baselines in machine translation.","key_machinery":"The load-bearing mechanism is the MQM taxonomy turned into a role-division scheme. Seven evaluator agents are each given one MQM category and a few-shot prompt that defines error subtypes and severity levels, and they independently return structured annotations. A single Editor agent then receives all annotations and revises the translation, applying a fixed priority rule: critical corrections first, major before minor, and no explanatory text in the final sentence. This machinery forces errors to be considered from separate perspectives, so an omission can be flagged by Accuracy, Terminology, and Style at once, while the editor's conflict-resolution rule keeps the final output a single coherent sentence. The paper credits the modular MQM-aligned roles, not just additional inference, for the observed improvement.","core_discovery":"On the paper's own terms, MAATS's core discovery is that MQM-category specialization is what unlocks better refinements: each evaluator agent sees only one slice of the error space, so the same base model, prompted separately as Accuracy, Fluency, Style, Terminology, Locale Convention, Audience Appropriateness, and Design and Markup agents, finds roughly 450% more translation issues than a single self-refinement agent, and the Editor agent's severity-priority rule converts those annotations into a corrected sentence. Across three base LLMs and eleven translation directions, MAATS achieves higher BLEU, METEOR, BLEURT, and COMET scores than both baselines, with significance concentrated in the neural metrics; a confusion-matrix comparison against human MQM labels shows more true positives and fewer false negatives, especially for Accuracy; and professional translators ranked MAATS first in an English-to-Chinese preference test. The paper reads these results as evidence that black-box LLM translation can be steered toward semantic and contextual fidelity rather than surface fluency by mirroring the roles of a human translation team.","pith_inferences":["Inference: the current comparison conflates architecture with inference budget, since the single-agent baseline runs one self-refinement pass while MAATS runs seven evaluator agents plus an editor; a budget-matched single-agent ablation is the natural next experiment and would tighten the causal claim.","Inference: the paper's own limitation section shows MQM lacks dimensions for emotional tone, irony, and rhetorical structure; adding an affective or discourse-level agent is a testable extension that could address the 'literature class hatred' failure the paper reports.","Inference: if the gain comes from independent error perspectives, the architecture should transfer to other generation tasks with structured quality dimensions, such as summarization or code review, where an editor can merge category-specific critiques."],"forward_implications":["MAATS is an inference-time architecture: with the same base LLM, users can get better translations by replacing one self-refinement pass with seven MQM-category evaluators plus an editor.","Because statistical significance is strongest for COMET and BLEURT and for distant language pairs, MAATS is most valuable where semantic fidelity matters and where single agents struggle most.","The MQM-aligned annotations make the refinement process auditable: each change can be traced to a category and severity, which supports translator review and quality-control workflows.","The paper demonstrates the architecture on eleven translation directions and three base LLMs, indicating that the gain is not tied to one model or one language pair."],"supporting_citations":[{"why":"This defines the MQM framework and scoring models that supply the error categories and severity levels used by the evaluator agents.","marker":"(Lommel et al., 2024)"},{"why":"This describes the systematic self-refinement method that MAATS uses as its single-agent baseline and contrasts with its own design.","marker":"(Feng et al., 2024)"},{"why":"This provides COMET, the neural metric where MAATS shows its most consistent statistically significant gains.","marker":"(Rei et al., 2020)"},{"why":"This supplies the human-labeled MQM reference data and confusion-matrix methodology used to validate MAATS's error detection.","marker":"(Freitag et al., 2021b)"},{"why":"This provides BLEU, the surface n-gram metric used in the automatic evaluation.","marker":"(Papineni et al., 2002)"},{"why":"This provides BLEURT, the neural metric used to measure semantic-level quality gains.","marker":"(Sellam et al., 2020)"},{"why":"This provides METEOR, another automatic metric in the evaluation battery.","marker":"(Banerjee and Lavie, 2005)"},{"why":"This supplies the positional voting method used to aggregate the professional translators' rankings of the three systems.","marker":"(Emerson, 2013)"},{"why":"This documents how LLMs amplify self-bias during self-refinement, which motivates the multi-agent design.","marker":"(Xu et al., 2024)"}],"fun_headline_variants":["MQM-specialized agents find 4.5x more translation errors","Seven specialized reviewers out-edit a single self-correcting LLM","450% more translation issues with multi-agent MQM review"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single-agent baseline is a fair comparison: it runs one self-refinement pass, while MAATS runs seven evaluator agents plus an editor, and the paper assumes those extra inference passes and the per-category division of labor do not by themselves explain the gains.","fun_headline_variants_meta":{"raw":{"variants":["MQM-specialized agents find 4.5x more translation errors","Seven specialized reviewers out-edit a single self-correcting LLM","450% more translation issues with multi-agent MQM review"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001291,"raw_usage":{"total_tokens":5270,"prompt_tokens":943,"completion_tokens":4327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":4267}},"tokens_in":559,"tokens_out":4327,"duration_ms":27959,"temperature":1.0,"reasoning_tokens":4267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:29:14.205968+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the single-agent baseline through multiple self-refinement rounds so it uses a comparable number of LLM calls per sentence as MAATS, then compare COMET, BLEURT, and human rankings on the same test sets. If the budget-matched single agent matches or beats MAATS, the improvement cannot be credited to the MQM multi-agent design.","supporting_citations":[],"review_version":1}