REVIEW
Tampered ML: Evaluating LLM Judgment of Machine Learning Research
Computation and Language · Submitted 23 Oct 2026 · 41 references
Pith's verdict: A careful, transparent benchmark for testing whether models can spot methodological flaws in ML experiments. The central finding holds up, though the human validation on the hardest items has the load-bearing soft spot the authors themselves flag.
T0 · AI review
Why Pith says this
Highlight any text to comment on it.
Referee report
The paper
The paper claims
Tampered ML benchmark shows frontier LLMs cannot reliably detect flawed ML experiment designs, achieving at most 78% accuracy on controlled pairwise comparisons.
The reading
What carries the argument
The Tampered ML benchmark: a pipeline that takes published ML experiments, extracts structured facts, rewrites them as self-contained proposals, then generates minimally edited counterfactuals that inject a single methodological flaw from an 8-category taxonomy (e.g., confounded comparison, blocked mechanism, measurement mismatch). Each item consists of a research context, original proposal, tampered proposal, and held-out flaw description. Models must select the more methodologically sound design in both presentation orders, enabling both per-order and both-orders accuracy metrics.
Extended reading notes
Core claim
The paper establishes that frontier LLMs, despite performing above chance, are not reliable judges of methodological soundness in ML experiment designs. In a controlled pairwise setting where one proposal is faithful to a published experiment and the other contains a single targeted flaw, the best model (Opus 5) achieves 77.9% both-orders accuracy (credit only if correct in both presentation orders). Models show strong position bias: they often select the original only when presented first. Qualitative analysis reveals that models substitute heuristics—treating specificity, transparency, or conventional practice as evidence of validity—even when these features are precisely what makes the tampered design flawed. Diagnostic interventions (flaw disclosure, category hints, review decomposition) recover many errors, indicating the models often recognize the flaw but fail to weight it decisively.
Load-bearing premise
The ground-truth labels are reliable: the injected flaws are real and consequential, and the original proposals are indeed more methodologically sound. The human validation, while supportive (86% validity estimated), is based on a single validator per item and overrepresents hard items, so if many unvalidated items have debatable flaws, the conclusion that models are unreliable would be overstated.
Editorial extensions
If this is right
- Current LLMs cannot be trusted to autonomously review or filter experiment proposals in alignment research or other scientific domains, as they miss even explicitly injected single flaws.
- Best-of-N selection and generate-and-filter pipelines that rely on LLM judgment will pass flawed designs and reject sound ones, potentially amplifying methodological errors.
- The finding that models reward specificity and conventional practice over genuine validity suggests that automated oversight may entrench standard but suboptimal methodologies.
- Diagnostic scaffolds (disclosing the flaw category) substantially improve performance, indicating that the benchmark measures not only flaw detection but also the ability to prioritize among multiple methodological concerns.
- The gap between pairwise and pointwise scoring (4–19 percentage points lower) implies that real-world performance, where models evaluate a single proposal, is even worse than the pairwise leaderboard suggests.
Reading between the lines
- The benchmark's construction relies on LLMs to generate and validate items, which introduces a circularity risk: if the generator models share blind spots with the evaluated models, the benchmark may underrepresent flaws that all current LLMs miss. The 86% human validation rate partially mitigates this but does not eliminate it.
- The position bias and confidence shifts suggest that models' reasoning chains are fragile and context-dependent; this has implications for interpretability: even when models produce correct rationales, the reasoning process may be inconsistent.
- If future models are trained on this benchmark (as warned by the authors), they may learn to detect the specific tamper patterns rather than general methodological reasoning. The benchmark should be used diagnostically, not as a training set for improving automated review.
- The failure mode where models accept a tampered design because it 'explicitly aligns with common practice' points to a deeper limitation: current LLMs lack a causal understanding of experimental validity and substitute surface-level norms.
Original abstract
One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.
The compiled record
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
The abstract, sentence by sentence
One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. Models must identify the more methodologically sound proposal in both presentation orders. A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. | background 1.00 | 0.05 | 0.03 | asserted 0.63 |
| s1 | Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including identifying flaws in research proposals and evaluating whether a proposed experiment supports its intended claim. | background 0.96 | 0.07 | 0.03 | asserted 0.99 |
| s2 | To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of an experiment proposal reconstructed from a published machine learning paper and a minimally edited counterfactual that introduces a targeted methodological flaw. | contribution 0.69 | 0.95 | 0.04 | asserted 0.99 |
| s3 | Models must identify the more methodologically sound proposal in both presentation orders. | method 0.99 | 0.32 | 0.04 | asserted 0.88 |
| s4 | A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81-91%). | method 0.56 | 0.95 | 0.05 | asserted 0.97 |
| s5 | Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). | result 1.00 | 0.92 | 0.55 | asserted 1.00 |
| s6 | They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional. | result 0.92 | 0.89 | 0.33 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
Figures
Showing 9 of the paper's 10 figures. See all figures in the PDF ↗ Figures reproduced from arXiv:2610.32114 by the authors.
References
Discussion
Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.