{"id":"7de6dafb-5643-4bb5-8cee-0e06454b75e6","arxiv_id":"2608.08143","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Frontier LLM judges show strong within-family agreement in a retrieval-augmented debate task, but that consensus does not reliably predict official human-annotation F1, with Quality showing the largest gap.","lead":"This paper tests whether six large language models can reliably score AI-generated debate responses. The models agree with each other, especially within the same family, but their consensus does not match official human annotations, most notably on response quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is tested through an unvalidated weak-supervision label model; the paper never directly checks whether observed multi-LLM agreement tracks official F1, so the reported misalignment may be a label-model artifact.","rationale":"The reader's weakest assumption concerns the validity of the official binary annotations as ground truth. That is related but not the load-bearing point: the paper's stated claim is about alignment with the official Subtask 2 F1, so by definition the official labels are the target; even noisy annotations would make the reported gap descriptive of the leaderboard rather than of evaluator truth. The load-bearing condition is that the consensus-based latent accuracy estimates, which do the work in Table 5, are a valid measure of multi-LLM evaluator agreement. The paper never establishes this. The FlyingSquid-style label model is built on voters that the paper itself shows to be non-independent, and the dependency-aware correction is not specified enough to audit. If the label model is misspecified, the reported Spearman correlations (0.366/0.224/0.500/0.011 in the expanded independent pool) and the 'largest gap on Quality' could be numerical artifacts. A direct correlation between observed agreement and official F1 would test the abstract's actual question without the latent-variable intermediation. This concern does not invalidate the paper's useful descriptive findings (same-family correlation structure, strict evaluator behavior, Quality difficulty), nor does it change the appropriate verdict: the paper should be conditionally accepted with a requirement to report the direct agreement-F1 analysis and to validate the label model. The current manuscript is honest about limitations and links code, which supports the conditional path.","tokens_in":9385,"tokens_out":10602,"duration_ms":109095,"concrete_test":"Compute, per Gricean maxim, the Spearman correlation between (a) system-level observed multi-LLM agreement, e.g., the mean pairwise correlation among the six frontier judges or the average judge score per system, and (b) official Subtask 2 F1, across the eight released systems, with permutation-based confidence intervals. Then validate the label model by correlating its per-judge latent accuracy estimates with each judge's directly computed accuracy (or F1) against the official binary annotations, using the same binarization threshold. If the direct agreement-F1 correlation is large (e.g., >0.7) or the latent accuracies fail to track observed official-label accuracies, the paper's negative conclusion is not established by the current analysis; if both remain near zero, the conclusion survives this check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 and Table 5 support the central claim with Spearman correlations between FlyingSquid-style latent judge-accuracy estimates and official Subtask 2 F1. These latent estimates are not observed agreement: they come from a label model that assumes voters are conditionally independent indicators of a latent binary label, includes ten non-LLM surface heuristics binarized at the median, and is patched with an underspecified 'dependency-aware' variant. The paper's own Figure 7 shows the independence assumption is violated exactly where it matters: same-family frontier LLMs are strongly correlated (Claude Opus/Sonnet mean r=0.844, GPT-4.1/4o r=0.772), and the added LLaMA-8B judges are weakly correlated with them. The dependency-aware modification is described only as 'explicitly linked' correlated voters, with no model specification, yet the headline numbers (e.g., Quality Spearman rising from 0.224 to 0.286) come from that modification. Because the latent accuracy estimates are never validated against each judge's observed accuracy on official labels, the reported low alignment could be an artifact of the label-model assumptions rather than a property of multi-LLM evaluator consensus. The abstract and conclusion phrase the claim in terms of evaluator agreement, but the evidence is about a model's inferred latent accuracies. The official annotations are the stated evaluation target, so their noisiness is not the central issue; the missing direct test is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends the DS@GT ARC working note for the Touché 2025 Retrieval-Augmented Debate task. It summarizes the official leaderboard results for six LLM-based response-generation and evaluation runs and then analyzes post-hoc whether multi-LLM evaluator consensus is a reliable proxy for official Subtask 2 F1. The analysis uses pairwise judge-score correlations, a FlyingSquid-style weak-supervision label model over LLM judges and surface heuristics, and compares latent judge-accuracy estimates with official F1 via Spearman correlation. The paper finds strong within-family judge agreement, weaker agreement with LLaMA-8B judges, and low Spearman correlations between latent accuracy estimates and official F1, with Quality showing a particularly large gap. The conclusion is that frontier LLM consensus does not reliably align with the official evaluation target.","tokens_in":9653,"tokens_out":7605,"duration_ms":70862,"significance":"If the finding is robust, it is a useful caution for shared-task evaluation: multi-LLM agreement is structured by model family and does not guarantee alignment with official human annotations. The paper's strengths include public code, use of an independent official target (not circular), and the explicit demonstration that within-family agreement inflates consensus. However, the central quantitative evidence passes through an unvalidated label model and a small sample, so the significance currently rests on the direct comparison in Figure 6 vs. Table 3 and on correlations that need further support.","major_comments":[{"comment":"The central claim is supported by Spearman correlations between latent judge-accuracy estimates and official Subtask 2 F1, but these latent estimates are outputs of a FlyingSquid-style label model that assumes conditional independence of voters; Figure 7 shows that assumption is violated for same-family frontier LLMs (e.g., Claude Opus/Sonnet mean correlation 0.844). Because the label model is never validated against observed judge accuracy on official labels, the reported low correlations could be artifacts of the model rather than properties of evaluator consensus. Please add a direct test: compute each judge's observed agreement or F1 against the official labels and correlate those observed quantities with official Subtask 2 F1, and also report the correlation between mean judge scores and official F1.","section":"Section 4.2 / Table 5"},{"comment":"The dependency-aware label model is described only as 'explicitly linked' correlated voters, with no specification of how dependencies are modeled or estimated. Since the dependency-aware Spearman values are headline results (e.g., Quality rises from 0.224 to 0.286), the model needs a full formal description or a reference to a concrete implementation; otherwise the numbers are not reproducible.","section":"Section 3.3 / Table 5"},{"comment":"The Spearman correlations are computed over only eight systems (six DS@GT runs, baseline, and SINAI). With n=8, none of the reported correlations is statistically distinguishable from zero at conventional levels; for example, ρ=0.580 for Relation has an approximate p-value around 0.13. Please report confidence intervals or permutation-test p-values, and discuss the power limitation.","section":"Table 5 / Section 4.2"},{"comment":"The claim that the largest gap is on Quality is supported by comparing latent accuracy average 0.687 with official Quality F1 average 0.244. These are different metrics (latent accuracy vs. F1) and no comparable gap is reported for the other maxims. Please report per-maxim mean latent accuracy and official F1 in the same table, or replace the 'largest gap' claim with a measure that is consistent across maxims.","section":"Section 5 / Section 4.2"}],"minor_comments":[{"comment":"The sentence 'We look at evaluator consensus through pairwise Pearson correlation computed over response turns separately per maxim and mean averaged' is unclear about what is averaged; specify whether the mean is over maxims or over judge pairs.","section":"Section 3.3"},{"comment":"The direct mismatch between averaged judge scores and official binary annotations is a key observation; consider quantifying it (e.g., rank correlation between mean judge score and official maxim-fulfillment proportion).","section":"Table 3 / Figure 6"},{"comment":"The number of rows (49,600) and the mapping from response turns to annotation rows should be explained, since 4,000 response turns would not obviously produce 49,600 rows with six judges and four maxims.","section":"Section 4.2"},{"comment":"Binarizing each voter at its median discards information; report sensitivity to the binarization threshold (e.g., using the mean or a fixed threshold such as 0.5).","section":"Section 3.3"},{"comment":"Table 2 reports internal evaluations from the working note; clarify whether these numbers are used later, since the analysis in Section 4.2 uses the organizer-released corpus.","section":"Section 4.1 / Table 2"},{"comment":"The paper states that the question is whether retrieval-augmented debate can be evaluated reliably through multi-LLM agreement; however, the same six LLMs serve as both generators and judges, and this self-evaluation aspect is not explicitly discussed as a limitation. Add a sentence acknowledging this potential bias.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task working-note extension with a useful negative result. The main risk is that the central quantitative claim is filtered through an unvalidated weak-supervision model; the direct comparison in Figure 6 vs. Table 3 is more convincing and should be elevated. The manuscript is within scope for the journal but needs the additional validation before the claim is accepted. There is some overlap with the authors' prior working note, but the post-hoc analysis appears to be new. Note that the reader's statement that the paper explicitly discusses the same-LLM-as-generator-and-judge limitation is not accurate; the text only says agreement should not be treated as independent evidence, so the limitation deserves explicit acknowledgment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core observation is worth knowing: six frontier LLM judges agree strongly within model families, but that consensus does not align well with the official Touché 2025 F1, with the largest gap on Quality. The within-family correlations (Claude 0.844, GPT 0.772) and the weak-supervision results are a useful caution for anyone using multi-LLM agreement as a proxy for evaluation quality in shared tasks.\n\nWhat the paper does well is mostly in the post-hoc analysis. It extends the working note with organizer-released data, explicitly discusses the circularity of using the same models as generators and judges, and ships code and data links. It also makes a fair point that adding lower-correlation LLaMA-8B judges changes rankings, which shows that judge diversity matters beyond raw strength. The discussion of Quality as a construct that collapses factuality, evidence support, and reasoning validity is sensible.\n\nNow the soft spots. The stress-test is partly right: the headline Spearman numbers in Table 5 (e.g., 0.224 for Quality) are correlations between latent judge-accuracy estimates from a FlyingSquid-style label model and official F1, not direct agreement between observed judge scores and official labels. The label model assumes conditional independence, and Figure 7 shows that assumption is violated for same-family judges. The dependency-aware variant is described only as \"explicitly linked\" with no model specification, so readers cannot reproduce the corrected numbers. That is a real transparency gap. However, the stress-test overstates when it says the paper never checks observed agreement against F1. Figure 6 directly compares averaged judge scores to official binary annotations and shows the DS@GT systems beat baseline under judge averages but not by the official metric, and Figure 1 gives a concrete example where all six frontier judges marked a response satisfactory while official annotators marked all four maxims unsatisfied. So the qualitative finding is not solely an artifact of the label model. The bigger issues are that the paper presents no significance tests on the Spearman/Kendall values (with only eight systems, the correlations are noisy) and the abstract overclaims that the systems are \"strong response generators\" when only two of six runs beat the baseline on official Subtask 1. It also does not assess annotation quality, but that is minor for a shared-task paper.\n\nVerdict: this deserves a serious referee. It is a diagnostic contribution for LLM-as-judge reliability and shared-task evaluation, and the honest limitations are a plus. The revision should either add a direct correlation between observed mean judge scores and official F1, validate latent accuracies against observed per-judge accuracy on official labels, and specify the dependency-aware model, or soften the claims to match what is actually shown. Do not desk reject.","headline":"The consensus-vs-F1 finding is likely real, but the headline numbers rest on an under-specified label model; still worth sending out.","tokens_in":10198,"tokens_out":3485,"would_cite":true,"duration_ms":37178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that six frontier LLM judges agree strongly on retrieval-augmented debate responses, but the agreement is largely a shared-training-data artifact and does not reliably track the official evaluation scores, with the…","keywords":["large language models","retrieval-augmented debate","Gricean maxims","LLM-as-judge","evaluator consensus","weak supervision","self-evaluation"],"falsifier":"Re-annotate a random sample of the evaluation turns with expert adjudicators, then recompute both official F1 and consensus-based latent accuracy; the claim weakens if the revised Quality labels bring the two measures into close agreement.","tokens_in":9145,"feed_emoji":"⚖️","tokens_out":8044,"duration_ms":73597,"temperature":0.7,"pith_summary":"The paper extends a shared-task working note to ask a methodological question: can the consensus of multiple LLM judges stand in for official evaluation in retrieval-augmented debate? It answers no. Six frontier LLMs from three providers agree strongly as judges, but the agreement is driven by model family and shared training data, and the consensus-based latent accuracy estimates do not reliably match the official Subtask 2 F1 scores. The largest disconnect is on the Quality maxim, where judges agree on surface plausibility without verifying factual support. Even a dependency-aware weak-supervision model closes only part of the gap. If the finding holds, retrieval-augmented debate evaluations cannot simply average a panel of LLM judges and expect the result to track the official target.","feed_headline":"LLM judge consensus misses official debate scores, worst on quality","feed_subtitle":"Six frontier models agree strongly, but the agreement comes from shared training data rather than from measuring debate quality.","key_machinery":"The central object is the multi-LLM judge panel: six frontier LLMs queried through a retrieval-augmented prompting pipeline, each producing scores from 0 to 1 on the four Gricean maxims. The decisive machinery is the pairwise correlation structure among judges, with same-family judges correlating around 0.84 and 0.77 for the two provider pairs, combined with a weak-supervision label model that treats binarized judge scores and surface heuristics as noisy votes and estimates latent accuracies for comparison against official F1. The misfit between latent accuracy and official F1, with the largest gap on Quality, carries the argument.","core_discovery":"The paper shows that when six frontier LLMs from three families serve as judges of retrieval-augmented debate responses, they agree strongly, especially within the same model family, and a weak-supervision label model over those judges recovers essentially the same consensus as the raw average. Yet that consensus barely tracks the official Subtask 2 F1 scores: dependency-aware latent accuracy averages 0.687 for Quality while the official Quality F1 averages 0.244, and Spearman correlations between latent accuracy and official F1 stay below 0.6 for all maxims, with Manner at 0.075. The authors conclude that high multi-LLM agreement is not a reliable proxy for the official evaluation target, with the largest gap on the Quality maxim.","pith_inferences":["If the official annotations themselves are noisy, the largest gap on Quality could reflect annotation problems as much as judge unreliability; adjudicating a sample of the roughly 4,000 evaluation turns would separate the two.","The same consensus-versus-target comparison could be run as a pre-submission sanity check in other shared tasks with binary labels: when same-family judges dominate, report family-specific correlations before trusting consensus.","A testable extension would split Quality into sub-scores such as factuality, evidence support, reasoning validity, and usefulness, and measure whether judge consensus then tracks official scores better, since the paper argues the collapsed single score invites surface-plausibility agreement."],"forward_implications":["A shared-task leaderboard cannot be safely reconstructed by averaging a panel of frontier LLM judges, because the consensus ranking would not match the official binary-annotation ranking.","Evaluators that draw on a single model family inherit a shared bias, so apparent panel agreement overstates confidence in the scores.","Weak-supervision calibration only changes rankings when the voter pool includes non-redundant judges; adding diverse small models shifts the rankings, while adding more frontier models does not.","For the Quality maxim, any evaluation that relies on multi-LLM agreement should be treated as measuring surface plausibility rather than factual, evidence-based quality."],"supporting_citations":[{"why":"Defines the shared task and the official binary-annotation evaluation protocol that supplies the F1 target.","marker":"[6]"},{"why":"Supplies the four Gricean maxims (Quantity, Quality, Relation, Manner) that structure both subtasks.","marker":"[3]"},{"why":"The original working note that described the six-model retrieval-augmented pipeline and the internal evaluation results.","marker":"[9]"},{"why":"Provides the weak-supervision label-model machinery used to estimate latent judge accuracies.","marker":"[2]"},{"why":"Motivates the weak-verifier aggregation approach behind the calibration analysis.","marker":"[10]"},{"why":"Describes the alternative open-weight judge pool whose less redundant signal changes the calibrated rankings.","marker":"[12]"}],"fun_headline_variants":["LLM judges agree strongly, but not on quality","Multi-LLM consensus isn't a reliable debate judge","Why LLM judge consensus misses official debate scores","Judge agreement fails quality check in debate tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The official binary annotations on the four Gricean maxims are treated as the ground-truth evaluation target; if those annotations are noisy or miss what debate quality really means, then the gap between LLM consensus and official scores shows disagreement with an imperfect label set rather than a failure of the LLM judges.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges agree strongly, but not on quality","Multi-LLM consensus isn't a reliable debate judge","Why LLM judge consensus misses official debate scores","Judge agreement fails quality check in debate tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2143,"prompt_tokens":874,"completion_tokens":1269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":1220}},"tokens_in":490,"tokens_out":1269,"duration_ms":10572,"temperature":1.0,"reasoning_tokens":1220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:20:16.672495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the evaluation turns with expert adjudicators, then recompute both official F1 and consensus-based latent accuracy; the claim weakens if the revised Quality labels bring the two measures into close agreement.","supporting_citations":[{"cited_title":"In: Carrillo-de-Albornoz, J., Gonzalo, J., Plaza, L., García Seco de Herrera, A., Mothe, J., Piroi, F., Rosso, P., Spina, D., Faggioli, G., Ferro, N","cited_arxiv_id":null,"evidence_quote":"Defines the shared task and the official binary-annotation evaluation protocol that supplies the F1 target."},{"cited_title":"In: Speech acts, pp","cited_arxiv_id":null,"evidence_quote":"Supplies the four Gricean maxims (Quantity, Quality, Relation, Manner) that structure both subtasks."},{"cited_title":"In: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2025)","cited_arxiv_id":null,"evidence_quote":"The original working note that described the six-model retrieval-augmented pipeline and the internal evaluation results."},{"cited_title":"In: III, H.D., Singh, A","cited_arxiv_id":null,"evidence_quote":"Provides the weak-supervision label-model machinery used to estimate latent judge accuracies."},{"cited_title":"In: Working Notes of the Confer- ence and Labs of the Evaluation Forum (CLEF 2025)","cited_arxiv_id":null,"evidence_quote":"Describes the alternative open-weight judge pool whose less redundant signal changes the calibrated rankings."}],"review_version":1}