{"id":"b5da7de1-7cc5-48ba-a16d-54ce59696a3e","arxiv_id":"2412.03597","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A review of known vulnerabilities in LLM benchmarks (contamination, overfitting, human and LLM judge bias) with a sketch of a proposed zero-day evaluation framework.","lead":"This paper reviews how LLM benchmarks can be hacked, contaminated, and biased, and argues that high leaderboard scores overstate real model ability. It proposes a future zero-day, zero-shot evaluation framework, but presents no new experiments or validated methods.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'pervasive vulnerabilities' claim rests on a narrative selection of prior critiques and on an unmeasured notion of 'true performance'; absent a systematic corpus and a benchmark-correlation test, the central overclaim is unsupported.","rationale":"I reviewed the paper in good faith. It is a survey and position paper, not an experimental study, and its central contribution is the claim that current benchmark scores systematically misrepresent LLM capability. For that claim to be true, two conditions must hold: (1) the sampled evidence of benchmark gaming must be representative of the evaluation ecosystem, and (2) there must be some benchmark-independent measure of 'true performance' against which leaderboard scores can be shown misleading. Neither is established. The paper's own text synthesizes known results, but the selection is ad hoc, with duplicated paragraphs and a truncated sentence in Section 3.7, consistent with an unfinished draft rather than a systematic analysis. The proposed zero-day, zero-shot framework in Section 5 is future work, not a validation of the central claim. The reader's weakest assumption—non-representative sampling—captures part of the problem, and I agree with that; my concrete test extends it by checking whether current leaderboard rankings actually diverge from an independent, contamination-controlled measure. If they do not diverge, the abstract's 'pervasive' and 'false perception' language is overclaimed. Therefore the reader's REJECT verdict stands, and no verdict adjustment is needed.","tokens_in":14947,"tokens_out":4619,"duration_ms":47310,"concrete_test":"Assemble a panel of 20-30 current instruction-tuned LLMs with published MMLU and GLUE-style scores; evaluate all of them on a contamination-controlled dynamic benchmark (e.g., LiveBench or a newly constructed private task suite) covering the same knowledge and reasoning domains; compute Spearman rank correlation and mean absolute score difference between leaderboard and held-out results. If rho > 0.9 and the mean absolute difference is small relative to the inter-model spread, the central 'false perception of progress' claim is not supported for the current generation; if rho is low or differences are large, the concern lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Sections 2-3 assert that vulnerabilities are 'pervasive across the evaluation spectrum' and that benchmarks create 'a false perception of progress.' The support is a curated set of examples—GPT-3 on LAMBADA and BERT on SQuAD in Section 3.1, human and LLM judge biases in Sections 3.6-3.10—with no systematic search strategy, inclusion criteria, or quantitative synthesis. The cited vulnerabilities involve mostly older models and specific tasks, yet the conclusion is a universal generalization about current GLUE and MMLU leaderboards. More importantly, the claim that models 'fail to demonstrate genuine language understanding' requires an independent operationalization of 'true performance'; the paper never provides one, so the conclusion is unfalsifiable as stated. If a held-out dynamic evaluation showed high rank correlation with MMLU and GLUE scores for current models, the 'false perception of progress' claim would be contradicted. The paper contains no such test. The weakest load-bearing step is therefore the unexamined representativeness of the selected studies combined with the absence of any benchmark-independent yardstick; both are needed for the central claim to hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a narrative review of vulnerabilities in large language model (LLM) evaluation, arguing that current benchmarks such as GLUE and MMLU are susceptible to gaming, data contamination, and evaluator bias, and that high leaderboard scores therefore create a false impression of genuine language understanding. After a brief history of NLP benchmarks, the paper surveys benchmark overfitting, public dataset leakage, test set contamination, task-specific optimization, adversarial benchmarking, human evaluation bias, and LLM-as-judge biases, with illustrative mathematical notation. It concludes that no single evaluation method is reliable and proposes future work on zero-day, zero-shot evaluation protocols and consortium-based dynamic benchmarking.","tokens_in":15130,"tokens_out":5463,"duration_ms":50859,"significance":"If the paper's central claim were established—that current benchmarks pervasively misrepresent LLM capabilities—it would have substantial implications for how the field measures progress and allocates trust in leaderboard results. The manuscript does collect a useful set of known critique themes, including contamination detection, annotation artifacts, self-preference in LLM judges, and human evaluator inconsistency. However, the paper provides no new experiments, no quantitative synthesis of existing findings, and no methodologically grounded framework for its 'systematic analysis' claim. Its value as a survey is further weakened by multiple misattributed citations and internal textual errors. The significance of the topic is high, but the contribution as written is a checklist of known issues rather than a substantiated systematic assessment.","major_comments":[{"comment":"The paper repeatedly claims a 'systematic analysis' and 'pervasive vulnerabilities across the evaluation spectrum', but no methodology is described anywhere in the manuscript. There is no search strategy, no inclusion or exclusion criteria, no corpus description, and no quantitative synthesis. The examples discussed (GPT-3 on LAMBADA, BERT on SQuAD, human evaluator and LLM-judge studies) are a curated selection, so the generalization to 'pervasive' vulnerabilities is unsupported. This is load-bearing because the central conclusion that benchmarks give a false perception of progress depends on the representativeness of the cited studies; without a defined method, the claim is an overgeneralization from an anecdotal sample.","section":"Abstract; Sections 2-3"},{"comment":"The mathematical formulations (B = P + ε, the contamination rate CR, the mutual information I(θ; D_test | D_train), and the exposure metric) are purely definitional and do not constitute evidence that any specific benchmark is inaccurate. The paper never estimates these quantities on actual models or datasets, so the equations provide no support for the claim that benchmarks misrepresent true performance. If the intent is to formalize known failure modes, this should be stated explicitly; as written, the formalism may give the misleading impression of a quantitative proof.","section":"Section 3.1, Eq. (1); Section 3.2; Section 3.3"},{"comment":"The central claim that benchmarks create 'a false perception of progress' requires a benchmark-independent operationalization of 'true performance,' which the paper never provides. Without such a yardstick, the claim is unfalsifiable as stated. A concrete test would be to compare leaderboard rankings with held-out dynamic evaluations or with performance on well-defined real-world tasks, and to measure the rank correlation; the paper contains no such test. Similarly, the assertion in Section 3.11 that 'models approach or achieve perfect scores on established benchmarks' is an empirical claim presented without data or citation.","section":"Section 3.11; Abstract"},{"comment":"Several citations do not support the claims to which they are attached, undermining the reliability of the survey. In Section 3.5, universal adversarial triggers are attributed to Wallace et al. with citation [6], but reference [6] is Raji and Buolamwini (2019), not Wallace et al. In Section 3.4, the finding that BERT models use shallow heuristics on MNLI is attributed to 'McCoy' with citation [56], but reference [56] is the MNLI dataset paper by Williams et al. (2018), not McCoy et al. In Section 3.1, the claims about GPT-3 on LAMBADA and BERT on SQuAD are supported by citations [25] and [20], which are the LAMBADA and SQuAD 2.0 dataset papers, not analyses of overfitting or pattern exploitation. These misattributions mean the cited evidence cannot be checked by the reader and weaken the manuscript's authority.","section":"Sections 3.4 and 3.5; References [6], [20], [56]"}],"minor_comments":[{"comment":"This section contains a duplicated paragraph: the text beginning 'One key limitation of human evaluations is the inherent inconsistency...' and ending '...poorly constructed [45]' appears twice verbatim, interrupting the flow of the argument.","section":"Section 3.6"},{"comment":"The section ends mid-sentence with 'as well as the exploration of entire' and no continuation; the sentence and the section appear to be truncated.","section":"Section 3.7"},{"comment":"The reference list has inconsistencies: reference [45] is used for two different papers (Chiang et al., 'Chatbot Arena', and Zheng et al., 'Judging LLM-as-a-Judge'), and references [18] and [41] both cite the same paper by Dubois et al. (2024). These duplicate entries need to be resolved.","section":"References"},{"comment":"The text refers to 'McCoy demonstrated' without a corresponding reference entry; the actual citation [56] points to the MNLI dataset paper rather than to any work by McCoy, so the intended source should be identified and cited correctly.","section":"Section 3.4"},{"comment":"The terminology 'Human-as-Judge' and 'LLM-as-Judge' is used with inconsistent capitalization and hyphenation; the authors should adopt a single consistent form throughout.","section":"Section 1.1"}],"recommendation":"reject","confidential_remarks":"The manuscript is a review-style paper with no new empirical contribution. Its central claim of a 'systematic analysis' and 'pervasive vulnerabilities' is not supported by the narrative selection of examples, and the absence of a benchmark-independent yardstick makes the main conclusion unfalsifiable. The citation errors are extensive enough that a full reference audit would be required, and the duplicated paragraph and truncated section suggest the paper is not ready for peer review. Even with substantial revision, the core issue remains that the advertised systematic analysis does not exist; the paper would need to be reframed as a position or survey paper with appropriately qualified claims, or supplemented with a genuine systematic methodology and empirical validation. Given the scope of changes required, rejection is the most appropriate outcome for this submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a competent but rough narrative survey of known benchmark vulnerabilities. It adds no new experiments, no systematic methodology, and no validated framework. The title and abstract promise a 'systematic analysis'; the body delivers a curated literature review with a few definitional equations. If you need a short overview of why static benchmarks, human judges, and LLM-as-judge are all gameable, this paper is a passable entry point. It accurately summarizes the findings it cites—contamination, overfitting, self-preference bias, evaluator inconsistency—and the citations generally point to the right primary sources.\n\nWhat the paper does well: the taxonomy of vulnerabilities (overfitting, contamination, adversarial examples, human bias, LLM-judge bias) is sensible, and the concrete examples (LAMBADA, SQuAD, Clark et al.'s human-evaluation contradiction, Zheng et al.'s self-preference numbers) are chosen appropriately. The math is almost all definitional notation for intuitive ideas, so it is harmless but not a contribution.\n\nSoft spots, in proportion: First, the central claim that current leaderboards create a 'false perception of progress' is never tested. The paper provides no benchmark-versus-out-of-distribution correlation, no held-out dynamic evaluation, no independent yardstick for 'true performance.' As stated, the conclusion is unfalsifiable. The stress-test note is right about this. Second, the 'systematic analysis' claim is unsupported: there is no search strategy, inclusion criteria, or synthesis method. Third, the draft is sloppy: Section 3.6 repeats the same paragraph nearly verbatim, Section 3.7 ends mid-sentence ('exploration of entire'), and citation [45] is used for two different works. These are fixable in revision, but they indicate the paper is not submission-ready. Fourth, the zero-day zero-shot framework in Section 5 is a sketch, not a method; no protocol, no metrics, no pilot.\n\nBottom line: this paper is for a reader new to the critique of LLM benchmarks who wants a quick map of the landscape. It does not advance the conversation, and the overclaiming and drafting problems would need substantial work. I would not send it to peer review in its current form. My advice: desk reject, but perhaps suggest the authors resubmit a properly systematic survey or a paper with an actual dynamic-evaluation experiment.","headline":"A readable but rough survey of known benchmark vulnerabilities; no new evidence or systematic method, so the advertised 'systematic analysis' overclaims.","tokens_in":15669,"tokens_out":2007,"would_cite":false,"duration_ms":19247,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that current LLM benchmarks, from GLUE and MMLU to human and LLM judges, can be gamed, so top scores do not prove genuine language understanding.","keywords":["large language models","benchmark hacking","data contamination","LLM-as-judge","human evaluation","leaderboard reliability","Goodhart's law","evaluation bias"],"falsifier":"Take the current top-ranked models on GLUE, MMLU, and a popular LLM-judge arena, run a contamination audit to remove any leaked examples, then test the same models on a freshly written, never-public set of analogous tasks in the same domains; if the rankings and scores stay nearly the same and the generalization gap $G(M)$ is near zero for most models, the paper's claim of pervasive benchmark vulnerability would be contradicted for those benchmarks.","tokens_in":14734,"feed_emoji":"🎯","tokens_out":6365,"duration_ms":275794,"temperature":0.7,"pith_summary":"The paper seeks to establish that leaderboard-style evaluation of large language models is systematically unreliable: models can post high scores on GLUE, MMLU, and similar benchmarks while lacking genuine language understanding or adaptability. It argues that benchmark overfitting, public test-set contamination, selective reporting, human-judge inconsistency, and LLM-as-judge biases all inflate apparent progress and can be exploited deliberately. If the paper is right, widely quoted benchmark numbers should be read as weak evidence of real capability, and evaluation must move toward dynamic, contamination-resistant, domain-specific protocols. The paper is a narrative review rather than a new experiment; its conclusion rests on assembling existing studies that document each vulnerability.","feed_headline":"LLM benchmarks are vulnerable: top scores can hide real gaps","feed_subtitle":"A review argues contamination, overfitting, and biased judges make leaderboard numbers unreliable measures of true capability.","key_machinery":"The load-bearing object is the benchmark score treated as an estimator of true performance, written as $B = P + \\varepsilon$, together with the generalization gap $G(M) = E[S(M,B_{\\text{test}})] - E[S(M,B_{\\text{real}})]$. The paper uses these formalizations, plus the contamination rate and the mutual-information measure of leakage, to make explicit the ways a score can rise while capability does not; the named principle underneath is Goodhart's law, that a measure ceases to be good once it becomes a target. The formalisms do the argumentative work of showing that benchmark optimization, contamination, and judge bias are not hypothetical but quantifiable distortions.","core_discovery":"The paper's central claim is that the evaluation ecosystem for LLMs contains a structural mismatch: the measures being optimized—static benchmark scores and judge ratings—are not the capabilities the field actually wants, namely robust understanding and adaptation to novel tasks. It identifies three families of exploitation: benchmark hacking (overfitting and task-specific optimization), data contamination (training/test overlap that inflates scores), and evaluator bias (humans and LLM judges whose judgments are noisy, format-sensitive, self-preferring, or otherwise gameable). The authors formalize these distortions with simple equations: benchmark score $B$ decomposes as $B = P + \\varepsilon$; contamination rate is $CR = |D_{\\text{train}} \\cap D_{\\text{test}}| / |D_{\\text{test}}|$; mutual information $I(\\theta; D_{\\text{test}} | D_{\\text{train}})$ signals leakage; and a generalization gap $G(M) = E[S(M,B_{\\text{test}})] - E[S(M,B_{\\text{real}})]$ quantifies the distance between benchmark success and real-world performance. They conclude that no single existing evaluation method is trustworthy on its own, that near-perfect benchmark scores deserve skepticism, and that future frameworks should be zero-day, zero-shot, domain-specific, and governed so they can be iterated before models overfit to them.","pith_inferences":["A consequence the authors leave implicit: new benchmarks should ship with a contamination audit and a preregistered refresh schedule before their numbers are treated as evidence of capability.","The same mechanisms would likely apply to code and mathematics benchmarks, where public solutions make contamination easier; the paper does not test this, but its logic predicts similar inflation.","The generalization-gap formalism suggests a cheap diagnostic the authors do not propose: report $G(M)$ alongside every published benchmark score so brittle leaders are visible."],"forward_implications":["Leaderboard rankings should be interpreted as upper-bound marketing claims rather than measurements of understanding, because reported scores can be inflated by contamination and overfitting.","Near-perfect or saturated scores on static benchmarks are weak evidence of progress; they may reflect exploitation of dataset artifacts rather than model competence.","Human evaluation cannot serve as an unbiased gold standard: annotators disagree on the same outputs and can be fooled by superficial fluency.","LLM-as-judge scores need debiasing: judges show self-preference, length bias, and sensitivity to prompt phrasing, so single-judge numbers are not stable.","Evaluation practice should shift toward dynamic, zero-day, domain-specific tasks with contamination audits and transparent reporting of methodology."],"supporting_citations":[{"why":"Supplies the GPT-3 benchmark-race example and the case of inflated LAMBADA performance due to training overlap.","marker":"[1]"},{"why":"Defines GLUE, the multi-task benchmark the paper cites as an early comprehensive framework now vulnerable to saturation and exploitation.","marker":"[10]"},{"why":"Defines MMLU, the 57-subject static benchmark whose near-perfect scores the paper treats as evidence of benchmark redundancy.","marker":"[12]"},{"why":"Documents sensitivity of LLM leaderboards to small changes, supporting the claim that benchmark scores are unreliable indicators of capability.","marker":"[21]"},{"why":"Introduces the exposure metric used to detect memorization and test-set contamination in trained models.","marker":"[26]"},{"why":"Supplies a black-box method for proving test-set contamination, used to show leakage inflates reported performance.","marker":"[39]"},{"why":"Provides distribution-level evidence of data contamination and evaluation malpractice in closed-source LLMs.","marker":"[40]"},{"why":"Documents LLM-as-judge self-preference bias, judge instability, and susceptibility to fine-tuning exploitation (Zheng et al.).","marker":"[45]"},{"why":"Shows human evaluators are inconsistent and often wrong when judging generated text, undermining human evaluation as a gold standard.","marker":"[46]"},{"why":"Reports LLM-as-judge reliability and its limitations: high correlation with human scores but prompt sensitivity and manipulability (Chiang and Lee).","marker":"[47]"}],"fun_headline_variants":["LLM benchmarks: glossy scores, hidden flaws","Benchmark gaming exposed in LLM evaluation review","Why top LLM test scores may not mean true smarts","LLM leaderboards: Contamination and biased judges under fire"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument that these vulnerabilities are pervasive depends on the unstated assumption that the studies the paper selects are a representative sample of LLM evaluation practice and that no significant counter-evidence was left out; the paper gives no systematic search strategy, inclusion criteria, or quantitative synthesis to support that.","fun_headline_variants_meta":{"raw":{"variants":["LLM benchmarks: glossy scores, hidden flaws","Benchmark gaming exposed in LLM evaluation review","Why top LLM test scores may not mean true smarts","LLM leaderboards: Contamination and biased judges under fire"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1418,"prompt_tokens":984,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":600,"tokens_out":434,"duration_ms":4394,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:57:56.701966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the current top-ranked models on GLUE, MMLU, and a popular LLM-judge arena, run a contamination audit to remove any leaked examples, then test the same models on a freshly written, never-public set of analogous tasks in the same domains; if the rankings and scores stay nearly the same and the generalization gap $G(M)$ is near zero for most models, the paper's claim of pervasive benchmark vulnerability would be contradicted for those benchmarks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-3 benchmark-race example and the case of inflated LAMBADA performance due to training overlap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines GLUE, the multi-task benchmark the paper cites as an early comprehensive framework now vulnerable to saturation and exploitation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines MMLU, the 57-subject static benchmark whose near-perfect scores the paper treats as evidence of benchmark redundancy."},{"cited_title":"Saiful Bari, and Haidar Khan","cited_arxiv_id":null,"evidence_quote":"Documents sensitivity of LLM leaderboards to small changes, supporting the claim that benchmark scores are unreliable indicators of capability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the exposure metric used to detect memorization and test-set contamination in trained models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows human evaluators are inconsistent and often wrong when judging generated text, undermining human evaluation as a gold standard."}],"review_version":1}