{"id":"32ad5a74-7146-4868-8357-93a67803acce","arxiv_id":"2411.16797","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A study measures how often GPT-4, Claude, LLaMA, and Gemini agree on PhD-level statistics questions, finding that Claude and GPT-4 produce questions with higher inter-model agreement, but the reliability metric relies on the question writer's own answer as the reference.","lead":"Four large language models answered each other's PhD-level statistics questions, and the researchers measured how often they picked the same answer. The study suggests that Claude and GPT-4 write clearer questions than Gemini and LLaMA, but it does not verify whether the agreed-upon answers are actually correct.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reliability metric R_i in Eq. 5 is circular: it scores reliability as agreement with the question generator's own intended answer, so the reported percentages do not establish correctness or enhanced reliability absent external ground truth.","rationale":"The reader's weakest_assumption—that R_i uses the generator's own intended answer as a correctness proxy—is exactly the load-bearing flaw. Every reliability percentage, every confidence interval, and the central conclusion depend on this unvalidated assumption. The paper provides no external ground truth, no human judgments, and no known-answer dataset, so the metric cannot distinguish genuine correctness from self-consistent shared error. The statistical analysis does not repair this: the chi-square test is misapplied to answer-letter frequencies rather than to agreement rates, and the bootstrap confidence intervals only quantify sampling variability in agreement, not validity of the reliability construct. The paper does include a useful descriptive dataset and a clear experimental protocol, but descriptive agreement among the models is not evidence that the consensus answer is reliable. I agree with the REJECT verdict; the central claim is not supported as stated. The proposed concrete test—external expert scoring of a subset—would directly test whether the consensus answer and R_i actually track correctness, and would settle whether the concern is fatal or merely an overstatement.","tokens_in":11247,"tokens_out":2405,"duration_ms":26832,"concrete_test":"Perform an external-ground-truth audit on a random 50-question subset of the released dataset (Amiri-Margavi 2024). Have two PhD-level statisticians independently solve each of the 50 questions with the same four answer options and record the correct answer; a third statistician adjudicates any disagreements. Then: (1) compute the true accuracy of the majority-vote consensus answer across the three answering models on this subset; (2) compare this with the paper's reported reliability percentages; (3) compute R_i as defined in Eq. 5 for each question and test whether R_i=1 is significantly associated with actual correctness using Fisher's exact test on a 2x2 contingency table. If consensus accuracy is substantially below the reported reliability rates, or if R_i shows no significant association with true correctness, the central claim is empirically falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that inter-model consensus enhances response reliability without ground truth—rests entirely on the reliability metric R_i defined in Eq. 5. R_i equals 1 only when the majority-vote consensus answer A_cons^i matches the question-generating LLM's own intended answer A_LLM-q_i. This is circular: the generator's answer is treated as a reference truth, but there is no justification that this answer is correct, especially when the other three models disagree with it. The paper itself concedes overlapping training data can produce correlated errors and that models may 'reinforce misinterpretation of each other,' which would inflate apparent consensus without improving correctness. Therefore the reported reliability percentages (77%–92%) measure alignment with the generator, not reliability in any accuracy sense. The chi-square test (Eq. 6) does not rescue the argument: it tests whether answer letters A–D are pooled uniformly across questions (expected frequency N*n_j/K = 75 per letter), not whether response agreement exceeds chance, so the tiny p-values do not demonstrate meaningful consensus or correctness. Without an external ground truth or human expert validation, the conclusion that collaborative dynamics 'significantly enhance response reliability' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a collaborative framework in which four LLMs (GPT-4-0125-preview, Meta-LLaMA-3-70B-Instruct, Claude-3-Opus, Gemini-1.5-Flash) generate and answer 100 PhD-level statistics multiple-choice questions, with the stated goal of improving answer reliability when ground truth is unavailable. The authors compute majority-vote consensus, introduce a reliability metric that checks alignment between the consensus answer and the question-generating model's own hidden answer, and report confidence intervals, chi-square tests, and Fleiss' kappa to quantify inter-model agreement. They conclude that collaborative dynamics significantly enhance response reliability and that Claude and GPT-4 generate higher-quality questions than LLaMA and Gemini. The central claim, however, rests on a reliability metric that treats the question generator's answer as a reference without external validation, and the statistical tests do not establish that observed agreement reflects correctness rather than correlated model behavior.","tokens_in":11628,"tokens_out":1501,"duration_ms":15610,"significance":"If the central claim were established, the paper would contribute a practical method for evaluating LLM outputs in domains without ground truth, which is a relevant problem for automated assessment and specialized knowledge validation. The authors have also made a concrete experimental artifact available (a GitHub repository) and have used machine-checkable statistical computations. However, the significance is severely limited by the fact that the reliability metric is not anchored to any external ground truth, human judgment, or known-answer dataset. As a result, the reported percentages (77%–92%) measure agreement with the question generator's own answer, not reliability in the sense of correctness. The paper's own limitation paragraph acknowledges that overlapping training data can produce correlated errors and that models may reinforce each other's misinterpretations, which directly undermines the inference from consensus to reliability. These issues are not presentation problems; they invalidate the paper's main conclusion.","major_comments":[{"comment":"The reliability metric R_i is defined as 1 only if the majority-vote consensus answer A_cons^i equals the question-generating LLM's own answer A_LLM-q_i. This is circular: the generator's answer is used as the reference for judging whether the consensus is correct, but no justification is given that the generator's answer is correct, especially when the other three models disagree. The paper's own limitation paragraph concedes that models can share overlapping training data, leading to correlated errors and 'reinforce misinterpretation of each other.' Consequently, the reported reliability percentages (e.g., Claude 92%, LLaMA 77%) measure alignment with the generator, not reliability in any accuracy sense. This is the load-bearing assumption of the paper and it is untested.","section":"Section 3, Eq. (5)"},{"comment":"The chi-square test is misapplied as evidence of meaningful consensus. The test in Eq. (6) compares the observed frequencies of answer choices A–D pooled across all questions and models against a uniform distribution, using expected frequency E_k = N * n_j / K = 75 per letter. A small p-value only shows that the answer choices are not uniformly distributed; it does not show that the models agree with each other or that their consensus is correct. For example, if all models always chose answer A for every question, the chi-square test would be significant, but that would indicate bias, not reliability. The tiny p-values in Table 4 therefore do not support the claim that inter-model agreement is meaningful or that consensus enhances reliability.","section":"Section 4, Eq. (6) and Table 4"},{"comment":"The confidence interval analysis is used to rank question-generation quality, but the intervals are computed for consensus rates (agreement among answering models), not for correctness. The conclusion states that 'Claude and GPT-4 emerged as the most reliable collaborators, achieving reliability percentages of 92% and 90%, respectively,' where reliability is defined by Eq. (5). Since the reference answer is the generator's own answer, the ranking of models as 'reliable' is entirely dependent on the circular metric. Without an external ground truth or human expert validation, the claim that 'collaborative dynamics among multiple LLMs can significantly enhance response reliability even in the absence of ground-truth data' is unsupported.","section":"Section 4, Table 3 and Conclusion"}],"minor_comments":[{"comment":"The notation in Eq. (6) defines O_k and E_k, but the expected frequency is written as E_k = N * n_j / K, which conflates the total number of responses (N * n_j) with the number of questions N. Clarify that the total observed count is across all response models, and ensure the degrees of freedom are stated correctly for the pooled test.","section":"Section 3, Eq. (6)"},{"comment":"The p-value for LLaMA is presented as '1 .67 × 10−10' with a spacing error; it should read '1.67 × 10−10'.","section":"Section 4, Table 4"},{"comment":"Equation (1), R = f(CI, DC, CF), is purely formal and is not used anywhere in the analysis. Either connect it to the actual reliability metric of Eq. (5) or remove it to avoid the impression of a non-operationalized theoretical framework.","section":"Section 1, Eq. (1)"},{"comment":"The text states that 'an answer is considered reliable if at least two other LLMs agree with the question-generating LLM's response,' which is a clear operational definition, but this definition highlights the circularity of the reliability metric. Please discuss explicitly why the generator's answer is a valid reference.","section":"Section 4, paragraph after Figure 2"}],"recommendation":"reject","confidential_remarks":"The paper's central claim is unsupported due to the circular reliability metric and the misapplication of chi-square testing. The authors acknowledge the correlated-error limitation themselves, which further weakens the inference from consensus to correctness. These are not fixable with local edits because the entire experimental design lacks an external anchor for reliability. A major revision would require either a new validation protocol with human experts, known-answer benchmarks, or at least a careful analysis showing that the generator's answer is trustworthy in a nontrivial sense. Given the current scope, rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a clean empirical description of how often four LLMs agree on each other's PhD-level stats MCQs, but its central claim — that inter-model consensus measures reliability without ground truth — does not survive contact with its own metric.\n\nWhat's new: a dataset of 100 questions generated by each of four models, with the other three answering, plus agreement statistics (Fleiss' kappa, bootstrap CIs) for these specific 2024 models. Role rotation and a public data link are good practice, and the descriptive finding that Claude- and GPT-4-generated questions yield higher inter-model agreement than LLaMA/Gemini is plausible and modestly interesting.\n\nThe soft spots are load-bearing. Equation (5) defines reliability as whether the majority answer equals the question generator's own answer. That is not reliability; it is self-consistency with the generator. The paper's own limitation paragraph concedes shared training data can produce correlated errors, which would inflate exactly this measure. So the 77–92% numbers do not establish correctness or 'enhanced reliability' in any accuracy sense. The chi-square test is also misapplied: Eq. (6) tests whether answer letters A–D are uniformly distributed across all responses, not whether agreement exceeds chance. Tiny p-values could just reflect option-preference bias. Fleiss' kappa is appropriate for measuring inter-rater agreement, so the descriptive layer is fine, but it cannot carry the conclusion. The conclusion literally states that collaborative dynamics 'significantly enhance response reliability' — that is unsupported.\n\nThe authors do list the right limitations, and the paper is not incoherent. A substantial revision that (a) reframes the claim as measuring agreement rather than correctness, and (b) validates against a known-answer benchmark or human expert labels would make this a useful contribution. As written, the central result is unsubstantiated.\n\nRecommendation: send it to review — the empirical data and question are worth engaging — but the referee should require the authors to address the circularity and reanalyze or remove the chi-square test. A serious referee would flag these issues quickly.","headline":"A tidy descriptive agreement study with a circular reliability metric and a misapplied chi-square test; the headline claim doesn't hold.","tokens_in":11980,"tokens_out":2289,"would_cite":false,"duration_ms":22674,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When four LLMs agree on an answer, that agreement can stand in for ground truth, this study claims.","keywords":["Large Language Models","Collaborative Intelligence","Inter-Model Consensus","Reliability","Question Generation","Majority Voting","Ground-Truth-Free Validation","Statistical Agreement"],"falsifier":"Take the same 100 questions and have human PhD-level statisticians mark the correct answers. If majority-vote consensus matches the generator's answer in most cases where the generator is wrong, or if consensus rates remain high on questions where the experts disagree with all models, the claim that agreement proxies reliability is falsified. The data are already public, so the comparison requires only expert labels.","tokens_in":11075,"feed_emoji":"🤝","tokens_out":4229,"duration_ms":37451,"temperature":0.7,"pith_summary":"This paper argues that when several large language models answer the same question independently, their agreement can serve as a reliability signal even when no correct answer is known. Four models took turns generating 100 PhD-level statistics multiple-choice questions; the other three answered, and the study measured how often they converged on one option. The authors claim that higher consensus tracks higher question quality and that the question-generating model shapes consensus: Claude and GPT-4 produced the clearest questions, while LLaMA produced the most ambiguous. If this is right, collaborative validation could substitute for expensive human ground truth in specialized domains. The practical payoff is a way to rank both answers and question generators without labelled data.","feed_headline":"LLM consensus can flag reliable answers with no ground truth","feed_subtitle":"On 100 PhD-level stats questions, agreement with the generator ranged from 77% to 92%.","key_machinery":"The load-bearing mechanism is the reliability metric $R_i$ of Eq. (5): for each question, $R_i = 1$ if the majority-vote consensus answer equals the answer $A^{\\mathrm{LLM\\text{-}q}}_i$ supplied by the question-generating model, and $0$ otherwise. This metric is paired with majority voting (Eq. 4), bootstrap confidence intervals on consensus rates, chi-square tests, and Fleiss' kappa. The metric is what turns the observation 'the models agree' into the conclusion 'the answer is reliable.'","core_discovery":"On the paper's own terms, the central discovery is that inter-model consensus is a usable proxy for answer reliability in the absence of ground truth. Using majority voting among three answering models, the authors define the consensus answer and a reliability score that checks whether the consensus matches the answer intended by the model that generated the question. Full agreement rates ranged from 86% for Claude-generated questions down to 65% for LLaMA-generated ones, and reliability scores ranged from 92% for Claude to 77% for LLaMA. Chi-square tests and Fleiss' kappa are offered as evidence that the observed agreement is not random. The conclusion drawn is that collaborative dynamics among multiple LLMs can significantly enhance response reliability even when no ground-truth answer is available.","pith_inferences":["The reliability metric equates 'agreement with the generator' with 'correctness', so it cannot distinguish a shared misconception from a shared truth; a known-answer benchmark would settle which.","Because the four models may share training data, correlated errors could inflate consensus; the paper's own limitation section concedes this possibility.","A testable extension is to replace the generator's intended answer with expert human labels on the same 100 questions and compare the reliability scores; if consensus frequently contradicts experts, the proxy fails.","Consensus rates could be compared across more diverse model families to test whether agreement is driven by architecture or by overlapping training data."],"forward_implications":["Consensus-based validation could be deployed in domains where labelled correct answers do not exist, such as graduate-level statistics or research review.","The question-generating model's identity becomes a quality lever: using a clearer generator raises downstream agreement among answerers.","Agreement statistics can flag ambiguous or poorly formulated questions before human review is needed.","Models with low consensus rates could be targeted for retraining or prompt adjustment to make their questions more interpretable."],"supporting_citations":[{"why":"Supplies the ensemble-learning rationale that combining multiple models improves reliability.","marker":"Dietterich [2000]"},{"why":"Provides the wisdom-of-model-crowds premise that consensus among models approximates correctness.","marker":"He et al. [2022]"},{"why":"Supports the claim that interacting decision-making agents can reach more reliable collective decisions.","marker":"Bahrami et al. [2010]"},{"why":"Defines GPT-4-0125-preview, one of the four models used as generator and answerer.","marker":"Achiam et al. [2023]"},{"why":"Defines Claude-3-Opus, the model that achieved the highest consensus and reliability scores.","marker":"Anthropic [2024]"},{"why":"Defines Gemini-1.5-Flash, one of the four models in the collaborative framework.","marker":"Team et al. [2023]"},{"why":"Defines Meta-LLaMA-3-70B-Instruct, the model with the lowest consensus and reliability scores.","marker":"Touvron et al. [2023]"},{"why":"Gives a directly related probabilistic consensus framework for LLM reliability that this study extends.","marker":"Naik [2024]"}],"fun_headline_variants":["Consensus among LLMs signals answer reliability without ground truth","When LLMs agree, trust the answer — no ground truth needed","Inter-model agreement marks reliable LLM answers, study finds","No ground truth? LLM consensus steps in as reliability gauge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes that the question-generating model's own intended answer is a valid reference for judging whether the consensus answer is correct; if that answer is arbitrary, biased, or wrong, the reliability scores do not measure reliability at all.","fun_headline_variants_meta":{"raw":{"variants":["Consensus among LLMs signals answer reliability without ground truth","When LLMs agree, trust the answer — no ground truth needed","Inter-model agreement marks reliable LLM answers, study finds","No ground truth? LLM consensus steps in as reliability gauge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1250,"prompt_tokens":866,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":314}},"tokens_in":482,"tokens_out":384,"duration_ms":4647,"temperature":1.0,"reasoning_tokens":314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:18:59.114655+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 100 questions and have human PhD-level statisticians mark the correct answers. If majority-vote consensus matches the generator's answer in most cases where the generator is wrong, or if consensus rates remain high on questions where the experts disagree with all models, the claim that agreement proxies reliability is falsified. The data are already public, so the comparison requires only expert labels.","supporting_citations":[],"review_version":1}