{"id":"eebae5e3-0312-4587-bf86-09af7d419f38","arxiv_id":"2411.18444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-agent, self-training LLM framework called MESA evaluates meeting summaries by detecting eight error types and reports higher correlation with human scores than existing automatic metrics.","lead":"This paper introduces MESA, a multi-LLM system that checks meeting summaries for eight error types, discusses possible errors among several AI agents, and adjusts its scoring using human feedback. It reports noticeably better agreement with human judges than standard metrics like ROUGE, BERTScore, or G-Eval, which matters because automatic quality checks are needed for AI-generated summaries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed average 0.25 gain from self-training is not yet trustworthy because the feedback report is derived from and evaluated on the same 170 annotated samples; a held-out or cross-validated evaluation is required.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: self-training feedback is generated from and evaluated on the same 170 annotated samples, with no held-out split. I agree with that diagnosis. The non-self-trained MESA variants (Single-0, Multi-0) provide some evidence for the architecture's value, so a full rejection is not warranted; but the abstract's headline average improvement and the 'self-training step helps align with human judgment' conclusion cannot be accepted on the present evidence. The proposed cross-validation check would settle whether the gain is genuine. Since the reader already assigned CONDITIONAL with this concern, my stress-test does not move the verdict, hence UNCHANGED. I note that the base-architecture comparisons, while also computed on the same small dataset, do not involve data-dependent prompt modification and are therefore less affected by the leakage concern. The lack of released code/data and the unresolved 170-vs-200 sample inconsistency are secondary but should be addressed in revision.","tokens_in":15645,"tokens_out":3417,"duration_ms":32352,"concrete_test":"Run a five-fold cross-validation of the self-training loop: for each fold, construct the per-error-type GPT-4 consolidated feedback report using only the human annotations in the other four folds, append that report to the MESA prompts, and score the held-out fold; average the point-biserial and Spearman correlations across folds for Single-1 and Multi-1. If the cross-validated averages are not materially above Single-0/Multi-0 (and above G-Eval-4), the self-training gain is in-sample rather than generalizable. As a companion check, resolve the 170-vs-200 sample discrepancy and report the exact N underlying every cell in Tables 3-5.","verdict_should_be":"UNCHANGED","load_bearing_attack":"MESA's self-training mechanism (Section 2.5) compares MESA's outputs with human annotations, has a GPT-4 judge consolidate per-error behavioral feedback, and appends that feedback to the prompts used for the three-step assessment. Section 3.1 then reports Single-1 and Multi-1 correlations on the same modified QMSum Mistake dataset, with no held-out split or cross-validation described. The comparison set is at most 170 samples, so the feedback report can be tuned to those exact samples; the observed ~0.1 average improvement of Single-1/Multi-1 over Single-0/Multi-0 (Section 3.2) may reflect in-sample alignment rather than a generalizable evaluation improvement. This matters because the headline claim of being 'on average 0.25 higher than previous methods' is carried by the self-trained variants, especially Multi-1 in Tables 3 and 4. The paper's own Limitations section acknowledges the dataset 'might seem small' and states 170 samples, while Appendix D.1 reports 200 meetings (169 erroneous); this unresolved sample-count inconsistency further weakens confidence in the exact evaluation setup. Without a train/test separation for the self-training feedback, the central claim of the abstract is unsupported for the self-trained configuration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents MESA, a multi-stage LLM evaluator for meeting summaries. MESA assesses each of eight error types through a single-aspect, three-step pipeline (candidate error identification, severity rating, scoring), optionally refines decisions through multi-agent discussion, and appends self-generated feedback derived from human annotations to its prompts. The authors evaluate on a modified QMSum Mistake dataset (reported as 170 samples in Section 2.1 and the Limitations section, but as 200 meetings in Appendix D.1) using ROUGE, BERTScore, G-Eval-4, and four MESA variants. They report higher point-biserial correlations for error presence and higher Spearman/Kendall correlations for error-impact severity, with average gains of roughly 0.2-0.25 over G-Eval-4, and conclude that MESA is a flexible, high-performing automatic evaluator.","tokens_in":15776,"tokens_out":5678,"duration_ms":52124,"significance":"If the reported gains generalize, MESA would be a practically useful contribution because automatic evaluation of meeting summaries remains difficult and current metrics correlate poorly with human judgment. The paper's strengths include a systematic component analysis (Tables 1 and 2), detailed prompts in Appendix A, a documented annotation process with inter-annotator agreement, and a clear statement that code and data will be released. The main limitation is that the self-training component is evaluated on the same annotations used to generate its feedback, so the headline improvements of the self-trained variants (Single-1 and Multi-1) are not yet supported as evidence of generalization.","major_comments":[{"comment":"The self-training mechanism in Section 2.5 uses human annotations to create feedback reports, and Section 3.1 reports the correlations of Single-1 and Multi-1 on the same modified QMSum Mistake dataset; no held-out split or cross-validation is described. Because the feedback is derived from the same 170 samples on which correlation is computed, the improvements of Single-1/Multi-1 over Single-0/Multi-0 (e.g., Table 4, OM: -0.27 to -0.42 for Single; -0.27 to -0.58 for Multi) may reflect fitting to the evaluation set. Please provide a held-out evaluation, such as generating feedback on a training subset and reporting correlations on a disjoint test subset, or leave-one-out cross-validation; alternatively, clearly relabel the self-training results as in-sample demonstrations rather than evidence of generalization.","section":"Section 2.5 / Section 3.1 / Tables 3-4"},{"comment":"The paper is inconsistent about the size of the evaluation set: Section 2.1 says the dataset 'comprising 170 samples,' the Limitations section repeats '170 samples,' but Appendix D.1 (Table 10) reports 'QMSum Mistake 200 (169)' meetings. This discrepancy affects the interpretation of all correlations and p-values; please reconcile the count and specify exactly how many summaries and annotations were used in each experiment.","section":"Section 2.1 / Limitations / Appendix D.1"},{"comment":"The averaged gains are reported as '0.1' and '0.15' in Section 3.2 and as '0.25' in the abstract, but no confidence intervals, standard errors, or multiple-comparison corrections are given for the correlations in Tables 3 and 4. Given the small sample size and eight error types, significance stars alone are insufficient to support the headline average improvement; please report uncertainty estimates or per-error-type effect sizes with confidence intervals, and define precisely how the aggregate gap is computed.","section":"Section 3.2 / Abstract"}],"minor_comments":[{"comment":"There are several typographical errors: Figure 1's caption uses 'singel-aspect' instead of 'single-aspect,' Section 2.3's heading uses 'Chalenge II' instead of 'Challenge II,' and Section 2.5 uses 'MESAarchitecture' instead of 'MESA architecture.'","section":"Figure 1 / Section 2.3 / Section 2.5"},{"comment":"The abbreviation for redundancy is inconsistent: RED is used in Section 2.1 and Table 9, while REP is used in Tables 2-5; please unify the notation.","section":"Section 2.1 / Tables 2-5 / Table 9"},{"comment":"The prompt templates state that responses must use double quotes in JSON, but the example output format in Figures 3 and 4 shows single-quoted keys and values; align the examples with the stated strict-format requirement.","section":"Appendix A"},{"comment":"The G-Eval-4 baseline is not fully specified; please report the exact prompt template, the number of runs, and the aggregation procedure, since LLM-based evaluation is known to be sensitive to sampling temperature and prompt wording.","section":"Section 3.1"},{"comment":"Table 1 reports balanced accuracy for the single-step and three-step approaches but does not include sample sizes or confidence intervals; adding these would help assess the stability of the reported improvements.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study for an industry-track NLP venue, but the self-training evaluation must be redone with a proper train/test separation before the central claim is credible. The base three-step single-aspect results are interesting and appear to support the value of the architecture; if the authors can show held-out gains for the self-trained variants, I would be supportive of publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a mixed bag: a genuinely useful architecture with a headline claim that does not survive contact with the data. The new piece is the three-step, single-aspect evaluation per error type (spot candidate instances, rate them, score the summary) plus multi-agent discussion and a self-training loop that appends LLM-generated feedback to the prompt. The component ablations are the strongest part. Moving from multi-aspect to single-aspect evaluation is well motivated, and the balanced accuracy gains in Table 1 are consistent. The non-self-trained variants (Single-0, Multi-0) beat G-Eval-4 across most error types in Tables 3 and 4, and that difference is the real finding. I would believe that result.\n\nThe soft spot is exactly where the reader and the stress-test put it. Section 2.5 derives the self-training feedback by comparing MESA's outputs with the human annotations, and Section 3.2 evaluates Single-1 and Multi-1 on that same set of 170 samples. No held-out split, no cross-validation. The ~0.1 average improvement from self-training is therefore plausibly in-sample alignment, and the abstract's \"on average 0.25 higher\" is carried by those self-trained variants. That is a load-bearing flaw in the headline, not a cosmetic one. The paper even acknowledges the dataset \"might seem small\" but then Appendix D.1 reports 200 meetings (169 erroneous), which does not match the 170 used in the experiments. That needs to be reconciled. Also, there are no confidence intervals on any of the correlations, only p-value stars. At n=170, that matters. And while the authors name AUTOCALIBRATE and FACTSCORE in related work, they do not compare against them.\n\nWhat holds up: the architecture is clearly described, the prompts are in the appendix, the annotation effort is real (Krippendorff's alpha 0.793), and the non-self-trained results are consistent. The paper is worth a serious referee, but the referee should demand a held-out or cross-validated evaluation of the self-training loop, public artifacts, and a resolution of the 170/200 inconsistency. I would send it back for revision rather than reject: the core idea is promising and the base results stand.\n\nMy verdict: conditional accept with major revision. The paper is for people building automatic evaluators for meeting summaries or other long-document NLG; they should read the component analysis and treat the self-training gain as unverified until it is shown to generalize.","headline":"The base three-step single-aspect architecture shows real, consistent gains over G-Eval-4, but the headline self-training improvement is evaluated on the same 170 samples that generated the feedback, so the 0.25 average claim does not yet hold.","tokens_in":16407,"tokens_out":3098,"would_cite":true,"duration_ms":28088,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage LLM pipeline out-correlates existing metrics on meeting summary quality by about 0.25.","keywords":["meeting summarization evaluation","LLM-based evaluation","multi-agent discussion","self-training","error detection","QMSum Mistake","human alignment","G-Eval"],"falsifier":"Run MESA on a held-out split of meeting summaries, for example 100 newly annotated summaries, and compare its Point-Biserial, Spearman, and Kendall correlations with human judgments against the correlations reported on the 170 samples used for self-training; if the correlations drop to the level of G-Eval-4 or below, the claimed 0.25 average improvement would be shown to be an artifact of feedback derived from the test samples.","tokens_in":15334,"feed_emoji":"🤖","tokens_out":8982,"duration_ms":67141,"temperature":0.7,"pith_summary":"This paper claims that an automatic evaluator can judge meeting summaries almost as reliably as human reviewers when the evaluation is broken into error-type-specific steps instead of a single overall score. The proposed MESA framework, built on GPT-4o, first identifies candidate error instances in a summary, then rates the severity of each candidate, then assigns a score to that error type, repeating this for eight error categories. A multi-agent discussion step refines the candidate lists and severity ratings, and a self-training loop adjusts the system's scoring behavior using a small set of human annotations. On a modified QMSum Mistake dataset of 170 meeting summaries, MESA's correlations with human judgments are, on average, 0.25 higher than those of previous methods. If this holds, meeting summaries could be quality-checked automatically without the cost and delays of human evaluation.","feed_headline":"MESA LLM evaluator beats prior metrics by 0.25 on meeting summaries","feed_subtitle":"Three-step error checks and self-training push automatic scoring closer to human reviewers.","key_machinery":"The load-bearing mechanism is MESA's three-step, single-aspect evaluation pipeline: an identification pass that lists candidate error instances with certainty scores, a severity pass that judges the existence and gravity of each instance, and a scoring pass that turns the observed errors into a Likert score per error type. This pipeline is wrapped in a multi-agent discussion protocol (a draft generator, three challengers, and a moderator) that refines the outputs, and a self-training loop in which an LLM judge compares MESA's reasoning and scores against human annotations, generating feedback appended to future prompts. The weighted combination of per-type scores (Equation 1) and the mapping to a quality score (Equation 2) carry the final rating.","core_discovery":"The central claim is that error-specific, three-step evaluation carried out by multiple LLM agents, followed by feedback-based self-training, makes an LLM-based evaluator substantially more aligned with human judgment than existing automatic metrics for meeting summaries. In MESA, each of eight error types is assessed separately: a first LLM pass lists potential error instances with chain-of-thought reasoning and confidence scores, a second pass rates each instance's severity and whether it is a real error, and a third pass converts the observed errors into a Likert score for that type. A multi-agent discussion protocol, with a draft generator, three challengers, and a moderator, refines the intermediate outputs and reduces score variance. The per-type scores are combined by a confidence-weighted sum, mapped to a 0–5 impact score and a 1–10 quality score, and a self-training loop uses an LLM judge to compare MESA's outputs with human annotations, appending the resulting feedback to the prompts. The authors report that with GPT-4o, MESA attains mid to high Point-Biserial correlations with human error detection and mid Spearman and Kendall correlations on error impact, on average 0.25 higher than prior methods.","pith_inferences":["Beyond the paper, the largest gain from self-training appears to be recalibration of score severity rather than better error detection; an implicit suggestion is that LLM evaluators are already strong detectors but systematically over-penalize, so prompt-level calibration may substitute for more expensive fine-tuning in other domains.","A natural extension is transferring the same three-step pipeline to other long-document genres, such as news or medical records, after replacing the error taxonomy; a quick probe would be to evaluate MESA on the original QMSum test set with human judgments.","An open risk the paper does not resolve is whether the self-training feedback memorizes the 170 annotated summaries; an explicit held-out evaluation would settle whether the 0.25 average improvement is real or an artifact.","The weighting scheme that prioritizes omission, hallucination, and irrelevance over readability errors encodes an implicit value judgment about what makes a meeting summary trustworthy; downstream users may want to expose those importance parameters as a tunable interface."],"forward_implications":["Meeting summary evaluation can move from costly human annotation to automatic, explanation-providing assessment at a fraction of the cost.","MESA's error-type-specific scores make it usable for feedback-based summary refinement, since each detected error instance is pinpointed in the text.","The self-training loop suggests that a small set of human annotations, on the order of 170 samples, can substantially reshape an LLM evaluator's scoring behavior for a new error taxonomy.","Because MESA accepts custom error guidelines, the framework should transfer to other summarization domains and to tasks with limited human-labeled data.","The single-model variant without multi-agent discussion offers a cheaper alternative with only a slight performance drop, making the approach feasible in lower-budget settings."],"supporting_citations":[{"why":"Supplies the QMSum Mistake dataset, the eight error definitions, and the feedback approach that MESA builds on.","marker":"Kirstein et al., 2024a"},{"why":"Defines G-Eval, the main LLM-based baseline whose correlation with human judgment MESA must beat.","marker":"Liu et al., 2023a"},{"why":"Provides the multi-agent debate protocol (MADP) used as MESA's discussion and refinement step.","marker":"Liang et al., 2023"},{"why":"The self-taught evaluator technique from which MESA's self-training loop is adapted.","marker":"Wang et al., 2024"},{"why":"The LLM-as-judge protocol used to generate the feedback reports during self-training.","marker":"Zheng et al., 2024"},{"why":"Chain-of-thought prompting, used in MESA's error-identification step.","marker":"Wei et al., 2024b"},{"why":"Documents error masking in existing metrics, motivating MESA's error-type-specific assessment.","marker":"Kirstein et al., 2024c"}],"fun_headline_variants":["MESA: multi-LLM evaluator lifts human correlation by 0.25","Three-step LLM judge matches humans better on meeting summaries","Self-training LLM evaluator beats prior metrics by 0.25","Multi-agent LLM evaluator closes gap with human review"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported improvement rests on the assumption that the self-training loop, which learned from the very same annotated summaries used to compute the reported correlations, transfers to summaries it has not seen.","fun_headline_variants_meta":{"raw":{"variants":["MESA: multi-LLM evaluator lifts human correlation by 0.25","Three-step LLM judge matches humans better on meeting summaries","Self-training LLM evaluator beats prior metrics by 0.25","Multi-agent LLM evaluator closes gap with human review"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000131,"raw_usage":{"total_tokens":1168,"prompt_tokens":1022,"completion_tokens":146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":70}},"tokens_in":638,"tokens_out":146,"duration_ms":1942,"temperature":1.0,"reasoning_tokens":70,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:12:08.738167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MESA on a held-out split of meeting summaries, for example 100 newly annotated summaries, and compare its Point-Biserial, Spearman, and Kendall correlations with human judgments against the correlations reported on the 170 samples used for self-training; if the correlations drop to the level of G-Eval-4 or below, the claimed 0.25 average improvement would be shown to be an artifact of feedback derived from the test samples.","supporting_citations":[],"review_version":1}