{"id":"9b882be3-64e6-4c9f-9db4-2a972b49b5e5","arxiv_id":"2507.19219","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"ArxivRoll builds one-time private benchmark questions from recent arXiv papers and computes a rugged score that it claims estimates contamination and training bias in public LLM benchmarks.","lead":"This paper introduces ArxivRoll, a benchmark that builds fresh test questions from recent arXiv papers and then measures how much LLM scores on older public benchmarks may be inflated. It is worth reading because it attempts the first quantitative estimate of how much public benchmark results overstate real LLM ability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RSI defined in Eq. (1) is not a proportion: the formula yields values above 1 (e.g., 1.21 in Table 3), and §2.3 concedes it is not normalized, contradicting the paper's central claim that RS measures the proportion of contamination.","rationale":"The reader identified the gap-attribution assumption as the weakest link. That is a serious concern, but there is an even more basic problem: the metric RSI does not mathematically represent a proportion even if the attribution problem were solved. The paper states in Section 2.3 that RSI is not normalized, yet the abstract and Section 1 call it a measure of the proportion of contamination. The reported values exceed 1, and Section 4.3 acknowledges values over 100%. Eq. (1) computes a symmetrized relative gap, not the share of public performance attributable to contamination. I therefore partially agree with the reader: the central claim is unsupported, but my reason is the internal inconsistency of the metric definition rather than only the absence of calibration. The private benchmark construction (SCP) may still have value as a dynamic evaluation tool, and the correlation experiments in Section 3.2 provide some evidence for that narrower claim. However, the load-bearing claim of quantifying contamination proportions is not supported by the paper's own equations and numbers. Verdict remains REJECT.","tokens_in":45701,"tokens_out":5348,"duration_ms":46118,"concrete_test":"Recompute RSI from Eq. (1) for the public/private accuracy pair corresponding to the Absolute RSI = 1.21 reported for Phi-1 in Table 3, and compare it with the contamination proportion (p−c)/p implied by the paper's interpretation. Because any proportion is bounded above by 1 and the reported RSI is 1.21, the inequality already falsifies the 'proportion' claim; no new experimental data are required. For a formal check, set p=1, c=0 in Eq. (1): RSI=2, while a proportion cannot exceed 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Rugged Scores 'quantify the proportion of cheating behavior' (Section 1) and that the public-private accuracy gap 'reflects the proportion of contamination in the model's performance on public benchmarks' (Section 2.1). This claim fails at the level of the metric definition, before any external calibration is considered. For a single matched pair, Eq. (1) reduces to RSI = 2(p−c)/(p+c), where p and c are public and private accuracies. This is a symmetrized relative gap, not a proportion: it is unbounded above and approaches 2 as c → 0. The paper's own Table 3 reports Absolute RSI values of 1.21 and 1.41, and Section 4.3 explicitly notes they are 'even larger than 100%'. A proportion must lie in [0,1]. Section 2.3 also concedes 'RSI is not a normalized metric'. Even granting the paper's unvalidated assumption that the entire gap is caused by contamination, the contamination proportion under that model would be (p−c)/p = 1 − c/p, not 2(p−c)/(p+c). The two differ for every nontrivial gap (e.g., p=0.5, c=0.25 gives RSI=0.67 vs. a true proportion of 0.5). Therefore the metric cannot quantify the proportion of contamination; the headline novelty of the paper is unsupported by its own definition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ArxivRoll, a dynamic LLM evaluation framework with two components: SCP, an automated generator of private test cases from recent arXiv articles, and Rugged Scores (RS), which are claimed to quantify the proportion of public benchmark contamination and training bias. The authors construct private benchmarks, meta-evaluate their quality, and use the public-private performance gap to compute RSI values for a range of open and closed LLMs. The central claim is that these rugged scores measure the proportion of cheating behavior or contamination in a model's performance on public benchmarks.","tokens_in":46008,"tokens_out":9355,"duration_ms":84596,"significance":"If the interpretation of RS were valid, the paper would offer a valuable, transparent, reproducible, and dynamically refreshed evaluation pipeline, and its quantification of overestimation would be a useful complement to static benchmarks. The SCP construction is genuinely interesting, the authors release their code, and the correlation analysis with ChatbotArena provides some evidence that the private benchmark ranks models sensibly. However, the central measurement claim is not supported: RSI as defined in Eq. (1) is not a proportion, the paper's own numbers exceed 100%, and no validation is provided that the public-private gap is attributable to contamination rather than to task-format differences.","major_comments":[{"comment":"The central claim that RSI quantifies the proportion of contamination is contradicted by the metric's own definition. For a single matched pair, the matched-pair term of Eq. (1) is 2(p−c)/(p+c), which is a symmetrized relative gap, not a proportion: it is unbounded above and approaches 2 as c → 0. Table 3 reports Absolute RSI values of 1.21 and 1.41, and Section 4.3 explicitly says these are 'even larger than 100%'. Section 2.3 itself concedes that 'RSI is not a normalized metric'. Even under the authors' assumption that the entire public-private gap is due to contamination, no standard contamination proportion equals 2(p−c)/(p+c); natural candidates such as (p−c)/p or (p−c)/(1−c) give different values (e.g., for p=0.5, c=0.25, the matched-pair term is 0.67 while these candidates are 0.5 and 0.33). The paper provides no derivation linking Eq. (1) to a proportion, so the headline claim that RS 'quantify the proportion of cheating behavior' is unsupported by the paper's own definition.","section":"§1, §2.1, §2.3, Eq. (1), Table 3"},{"comment":"The interpretation of the public-private performance gap as a contamination proportion requires that the gap be attributable to contamination rather than to differences in task format, content, difficulty, or domain shift. The public benchmarks (MMLU, MMLU Pro, and the vaguely described 'domain-specific datasets' in Appendix A) differ substantially from the private SCP tasks, which are sequencing, cloze, and prediction on arXiv text fragments. The paper provides no synthetic contamination experiment, no controlled benchmarks matched in format and difficulty, and no calibration against known contamination. The correlation analysis in Section 3.2 validates only that private-benchmark rankings resemble ChatbotArena rankings; it does not validate the gap interpretation. Consequently, Eq. (1) is at best a task-specific accuracy gap, and the contamination reading is an unvalidated assumption.","section":"§2.1, §3.2, §4.3"},{"comment":"The construction of the public-private pairs used in Eq. (1) is underspecified. Appendix A states that MMLU, MMLU Pro, and 'domain-specific datasets, such as those commonly used in Math' are used, but it does not report which public benchmarks are paired with which of the eight private domains, the number of pairs Np, the sizes of the unmatched sets, or how the aggregate terms in Eq. (1) are weighted. Since RSI depends on all of these choices, the RSI values in Table 3 are not reproducible from the manuscript as written. The authors should provide the exact benchmark inventory, the pairing rule, and the uncertainty propagated through Eq. (1).","section":"Appendix A, §2.3"},{"comment":"RSI values in Table 3 are reported without confidence intervals, despite the large standard errors in the underlying private-benchmark accuracies (e.g., several models in Table 2 have error bars of ±7–8 percentage points). For reported differences such as Llama3.1-70B (RSI 0.48) versus Llama3-8B (RSI 0.74), the gap is likely within the measurement noise. Without uncertainty propagation, the qualitative claims in Section 4.3 about which model families exhibit the highest overestimation are not supported.","section":"Table 3, Tables 2, 6, 7"}],"minor_comments":[{"comment":"The free parameters N and Nf are only defined in Appendix A (N=1, Nf=80); they should be defined in the main text where the SCP algorithm is introduced, and a sensitivity analysis for these choices would strengthen the benchmark construction.","section":"§2.2, Appendix A"},{"comment":"The axis labels in Figure 5 are garbled by font-encoding artifacts and are not readable, so the reader cannot verify the 'growing overestimation' claim from the figure.","section":"Figure 5"},{"comment":"The column headed 'Relative Rank Changes' mixes arrow glyphs with numeric values and is never clearly defined; it should be renamed and explained, for example by stating whether it is a relative RSI score or a rank shift.","section":"Table 3"},{"comment":"In the ArxivRollBench2026a results, 'moonshotai/kimi-k2.6' is ranked first with a valid accuracy of 70.8% despite a raw accuracy of 5.6% and a coverage of 7.9%; the pooled valid-response protocol can materially reorder the leaderboard when coverage is low, and this should be discussed in the main text.","section":"Appendix C, Table 13"},{"comment":"The title's phrase 'How Much Do Large Language Model Cheat on Evaluation?' suggests intentionality, while the paper also addresses unintentional contamination and biased overtraining; a more neutral phrasing such as 'overestimation in evaluation' would better match the content.","section":"§1"}],"recommendation":"reject","confidential_remarks":"The central measurement claim is internally inconsistent with the metric definition, so the paper cannot be accepted in its current form. The SCP benchmark infrastructure and the dynamic evaluation idea may be worth developing further, but re-framing the rugged score as a validated gap measure rather than a contamination proportion would require a substantial rewrite, not a local revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the SCP generator is a genuinely useful piece of engineering — fresh arXiv papers turned into sequencing/cloze/prediction items, with a stability check and reasonable correlations against ChatbotArena — but the Rugged Score is not a proportion of contamination. The paper’s own equations and tables sink the central claim.\n\nWhat’s good: ArxivRollBench is a real attempt at a dynamic, reproducibility-friendly private benchmark. Generating from recent papers with three objective task formats is sensible, the 32-seed stability test is honest, and the correlations in Table 1 (up to ~0.7 with ChatbotArena) support the claim that the benchmark measures something like general capability. Code and leaderboard are released. That is real value.\n\nWhere it breaks: the stress-test note is right. For a matched pair, RSI = 2(p−c)/(p+c). That is a symmetrized relative gap, not a proportion. It is unbounded, and Table 3 shows values like 1.21 and 1.41; Section 4.3 admits they exceed 100%. Section 2.3 concedes “RSI is not a normalized metric,” which directly contradicts the abstract and Section 1 calling it a proportion. Even granting the huge assumption that the entire public-private gap is contamination, the proportion under that model would be (p−c)/p, not 2(p−c)/(p+c). For p=0.5, c=0.25, RSI=0.67 but the contamination proportion would be 0.5. So the stated novelty — “first to measure the proportion of overestimation” — is unsupported. There is also no calibration experiment against known contamination; the interpretation is asserted, not tested. The RSII formula in Eq. (2) also looks wrong: it is a sum of squared deviations without dividing by N, so it is not a standard deviation, and the reported values are oddly small.\n\nReframe and it could be useful: call RSI a “ruggedness gap” or “overestimation index” with clear caveats about task-format differences, and it becomes a reasonable descriptive statistic. As written, the load-bearing interpretation is a claim without derivation.\n\nRecommendation: this deserves a serious referee — the engineering is solid and the flaw is fixable — but the current version should be rejected until the metric is redefined, renamed, or calibrated against known contamination.","headline":"Worth reading for the fresh-benchmark machinery, but the headline metric is not a proportion and the paper never validates it as one.","tokens_in":46530,"tokens_out":2563,"would_cite":false,"duration_ms":24524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ArxivRoll claims that the gap between a model's public and private benchmark scores directly measures the share of its performance that comes from contamination or training bias.","keywords":["LLM evaluation","benchmark contamination","overestimation quantification","one-time pad","private benchmarks","cloze tasks","sequencing tasks","Rugged Scores"],"falsifier":"Fine-tune a small open model on a held-out sample of ArxivRollBench while a matched control model sees none of it; if the fine-tuned model's RSI does not clearly exceed the control's, the gap is not mostly contamination.","tokens_in":45497,"feed_emoji":"📊","tokens_out":5553,"duration_ms":55204,"temperature":0.7,"pith_summary":"ArxivRoll tries to make overestimation in LLM evaluation measurable rather than just suspected. The framework builds private test sets from very recent preprint articles, keeps them secret until evaluation time, and treats them as one-time-use secrets in the style of a one-time pad. The Rugged Scores then take the normalised difference between a model's public and private scores in the same domain (RSI) and the spread of its private scores across domains (RSII), reading both as estimates of how much of the public score is inflated. If the paper is right, a leaderboard can report not only which model is strongest but how much of each score is real. The paper reports that several open-source model families show substantial RSI and that private benchmarks correlate with human-preference rankings.","feed_headline":"A new score detects how much LLMs cheat on benchmarks","feed_subtitle":"The two Rugged Scores estimate how much of each model's public score is real.","key_machinery":"The carrying objects are SCP and the Rugged Scores. SCP turns a short excerpt of a recent preprint into three objective multiple-choice tasks: Sequencing asks for the order of shuffled sentences, Cloze asks for the correct masked sentence in a paragraph, and Prediction asks which of four continuations follows the excerpt. This generator provides the confidentiality and freshness of the private set without human annotation. The Rugged Score RSI is the ratio-symmetric public-minus-private gap defined in Eq. (1), and RSII is the variance-based spread across private domains defined in Eq. (2). Together with the release-then-expire lifecycle of each private set, these form the mechanism by which the paper converts a suspected bias into a number.","core_discovery":"The paper's central claim is that overestimation is not a vague risk but a quantity, and that the quantity is the normalised gap between performance on public and private benchmarks. Concretely, RSI is defined by Eq. (1) as twice the average of $(M_p - M_c)/(M_p + M_c)$ over matched public–private pairs plus an analogous unmatched term, so RSI $=0$ means a model performs equally on both and higher RSI means its public score is increasingly inflated. RSII, defined in Eq. (2), is the standard deviation of the model's accuracy across private domains, with a normalised variant dividing by the mean, and is claimed to measure biased overtraining. The private benchmarks are generated automatically from papers too recent for training data, which is what makes the gap interpretable as contamination rather than as ordinary difficulty.","pith_inferences":["A natural test of the contamination reading is to take a small open model, fine-tune it on a held-out slice of ArxivRollBench, and show that its RSI rises while a matched control model's RSI stays flat; the paper's Eq. (1) predicts that directly.","The same ratio-symmetric gap could be computed between two versions of one benchmark, an old public release and a refreshed one, giving a contamination estimate that needs no new task generator.","For closed API models, the 'private' side is fresh but the public side is uncertain, so RSI should be interpreted as an upper bound on contamination rather than an exact proportion until the benchmark-pair difficulty is matched.","If SCP tasks measure mainly text coherence rather than domain knowledge, RSI would partly reflect task-format sensitivity; comparing RSI across SCP variants would reveal how much of the gap is format-specific."],"forward_implications":["A leaderboard that lists RSI next to accuracy lets a reader see at a glance whether a model's ranking comes from public-benchmark inflation or genuine ability.","Because every evaluation period uses a fresh private set, earlier ArxivRollBench releases can be published freely without weakening later assessments.","Domain-level RSI identifies which fields a model has overtrained on, so a developer can target training away from benchmark-specific strengths.","RSII gives a second, orthogonal signal: a model with balanced capabilities across domains scores low, while one trained hard on only the evaluated domains scores high.","Existing public benchmarks can be retrofitted as the 'public' side of the pair, so the framework does not require abandoning established benchmark sets."],"supporting_citations":[{"why":"Supplies the ChatbotArena human-preference benchmark used in Table 1 to validate that ArxivRollBench rankings correlate with an established, harder-to-fool evaluation.","marker":"Chiang et al. 2024"},{"why":"Provides the one-time-pad confidentiality concept that the framework adapts into fresh, one-use private benchmarks.","marker":"Shannon 1949"},{"why":"Demonstrates that fine-tuning on benchmark samples can inflate reported performance, motivating the need to measure overestimation.","marker":"Yang et al. 2023"},{"why":"Documents data contamination in LLM training, which is the public-benchmark failure mode the Rugged Scores are designed to quantify.","marker":"Dong et al. 2024"},{"why":"Provides the symbolic-formatting precedent that SCP extends to generate fresh private test cases across many domains.","marker":"Mirzadeh et al. 2024"},{"why":"Supplies the LM Evaluation Harness used to run the model evaluations reported in the paper.","marker":"Gao et al. 2024"}],"fun_headline_variants":["How much do LLMs cheat? ArxivRoll puts a number on it","ArxivRoll quantifies how much LLM benchmark scores are inflated","Rugged Scores measure LLM overestimation from contaminated benchmarks","ArxivRoll exposes the true share of LLM benchmark scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole contamination reading rests on the assumption that the public-minus-private performance gap is caused by contamination or training bias, not by the private tasks being harder, differently formatted, or drawn from a different distribution.","fun_headline_variants_meta":{"raw":{"variants":["How much do LLMs cheat? ArxivRoll puts a number on it","ArxivRoll quantifies how much LLM benchmark scores are inflated","Rugged Scores measure LLM overestimation from contaminated benchmarks","ArxivRoll exposes the true share of LLM benchmark scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2694,"prompt_tokens":976,"completion_tokens":1718,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1642}},"tokens_in":592,"tokens_out":1718,"duration_ms":12648,"temperature":1.0,"reasoning_tokens":1642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:57:26.168022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune a small open model on a held-out sample of ArxivRollBench while a matched control model sees none of it; if the fine-tuned model's RSI does not clearly exceed the control's, the gap is not mostly contamination.","supporting_citations":[{"cited_title":"N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M","cited_arxiv_id":null,"evidence_quote":"Supplies the ChatbotArena human-preference benchmark used in Table 1 to validate that ArxivRollBench rankings correlate with an established, harder-to-fool evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the one-time-pad confidentiality concept that the framework adapts into fresh, one-use private benchmarks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents data contamination in LLM training, which is the public-benchmark failure mode the Rugged Scores are designed to quantify."}],"review_version":1}