REVIEW 3 major objections 3 minor
Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that a programmatic variation of Putnam problems cuts o1-preview's accuracy from 41.9% to 22.3%, suggesting that high benchmark scores largely reflect memorization rather than mathematical reasoning.
desk verdict A useful new Putnam benchmark and variation protocol, but the memorization claim rides on an unvalidated equal-difficulty assumption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Variation protocol: a programmatic procedure that rewrites a competition problem by changing its variables, constants, and other surface elements while preserving its mathematical difficulty, generating an unlimited stream of fresh instances that are unlikely to have appeared in training data. Paired with the original problems, these variations isolate memorization from reasoning: if a model's accuracy collapses when only surface details change, the original score was probably memorized. The second piece of machinery is Teacher-Forced Accuracy (TFA), a metric that scores the model's generated reasoning trace token-by-token against a reference trace, extending evaluation beyond whether the final boxed answer is correct.
What would settle it
Have human Putnam contestants solve the 100 variations under exam conditions and compare their accuracy with the originals; a similar human drop would indicate a difficulty shift rather than memorization.
Extended reading notes
Core claim
Putnam-AXIOM is a benchmark of 522 original Putnam problems; Putnam-AXIOM Variation consists of 100 paired instances generated by programmatically perturbing variables and constants. The central empirical finding is that accuracy on the paired variations is far below accuracy on the originals: o1-preview scores 41.9% on the originals and 22.3% on the variations, a 19.6-point drop (46.8% relative), and all nineteen evaluated models show the same downward trend, with ten having non-overlapping 95% confidence intervals. The authors interpret this gap as suggesting memorization of original problems rather than genuine reasoning, and present the variation protocol as a contamination-resilient evaluation method producing an unlimited stream of equally difficult, unseen instances. They also propose Teacher-Forced Accuracy (TFA), a lightweight metric that evaluates the model's reasoning trace directly, as a complement to final-answer accuracy.
Load-bearing premise
The claim stands on the assumption that perturbing variables and constants yields problems equally as hard as the originals, so that the accuracy drop isolates memorization rather than a change in difficulty or format.
Editorial extensions
If this is right
- Scores on fixed math benchmarks should be read as upper bounds on reasoning ability, since even strong models lose nearly half their accuracy on perturbed variants.
- Programmatic variation is a viable, contamination-resilient benchmark design that can generate unlimited test instances without human annotation.
- Teacher-Forced Accuracy offers a way to evaluate reasoning traces automatically, not just final answers, making proof-style evaluation scalable.
- The 19.6-point gap on o1-preview is a concrete measure of the performance the paper attributes to memorization on this benchmark.
- Future model evaluations should include both originals and variations to distinguish reasoning improvements from recall of fixed problems.
Reading between the lines
- If the equal-difficulty assumption holds, the same protocol could serve as a contamination probe for any competition-style benchmark: a large originals-minus-variations gap becomes a measurable memorization index, and publishers could set a threshold above which a model's official score is suspect.
- The variation approach could transfer beyond math, for example to code benchmarks (renaming variables, reordering functions) or scientific reasoning, wherever fixed problem sets are at risk of contamination.
- A human baseline on the variations would be the most direct check: if human Putnam solvers also drop by a similar amount, the gap is difficulty, not memorization; if they do not, the paper's interpretation is strongly supported.
- Teacher-Forced Accuracy inherits the usual risks of reference-based scoring: a model could produce a shorter or differently worded but correct proof and be penalized, so it may need semantic equivalence checks to serve as a standalone metric.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Putnam-AXIOM, a benchmark of 522 Putnam Competition problems, and a companion set of 100 programmatically perturbed 'Variation' problems. The central claim is that model accuracy drops sharply on the paired Variations relative to the Originals (o1-preview: 41.9% to 22.3%), and that this gap 'suggests memorization' and demonstrates the need for dynamic, contamination-resilient benchmarks. The abstract also proposes Teacher-Forced Accuracy (TFA), a metric for directly scoring reasoning traces. This review is based on the abstract only, as the full text was not available.
Significance. If the central claim were validated, the benchmark would be a valuable contribution: it is a public, functional-variation benchmark aimed at a genuine problem (training-set contamination), and the TFA metric could be useful for automating proof evaluation. The abstract reports concrete numbers with confidence intervals, and the code/data are promised publicly. However, the significance hinges entirely on the unverified premise that the variations are 'equally difficult' to the originals. Without a human baseline or independent difficulty calibration, the observed performance drop is equally consistent with a difficulty shift induced by the perturbation protocol. The contribution is therefore potentially significant but currently under-evidenced.
major comments (3)
- [Abstract, 'equally difficult' claim] The abstract asserts that the variation protocol produces 'equally difficult, unseen instances' and that the accuracy drop on the Variations 'suggests memorization,' but no evidence is provided that the perturbations preserve difficulty. Changing variables and constants can affect the need for case analysis, the size of intermediate arithmetic, the availability of simplifying identities, and the probability that a shortcut succeeds. The 19.6% drop for o1-preview and the similar trend across other models are consistent with a difficulty shift or a format change, not necessarily memorization. To support the memorization interpretation, the paper must validate difficulty equivalence, for example by showing that human solvers achieve comparable accuracy or solving time on both sets, or by matching items on an independent difficulty proxy.
- [Abstract, contamination-resilience claim] The abstract says the benchmark is 'contamination-resilient' and that the performance gap 'suggests memorization,' but no contamination test is described. A direct test would be needed, such as checking n-gram overlap between the training data and the Variation prompts, probing the model with near-duplicates, or comparing behavior on semantically identical problems with different surface forms. Absent such a test, the drop could reflect a distribution shift unrelated to memorization, such as changed notation, prompt formatting, or problem presentation. The claim that the variation protocol 'removes' memorization while preserving reasoning difficulty is the load-bearing premise and cannot be inferred from the performance gap alone.
- [Abstract, TFA metric] The abstract introduces Teacher-Forced Accuracy (TFA) as a metric that 'directly scores reasoning traces and automates natural language proof evaluations,' but gives no description of how the metric works, how it handles partial credit, or how it was validated. Because TFA is presented as a companion contribution, the paper should specify the exact scoring mechanism and compare it against human evaluation or other proof-grading metrics; otherwise it is difficult to assess whether TFA adds value beyond boxed-answer accuracy.
minor comments (3)
- [Abstract, wording] The phrase 'These gaps suggest memorization' is too assertive for an observational performance gap; it should be qualified as 'are consistent with memorization' or similar, given the alternative explanations discussed above.
- [Abstract, dataset sizes] The abstract reports 522 original problems and 100 variations; the large size difference is not discussed, and it is unclear whether the paired comparison uses all 522 originals or only the 100 that were varied. This should be clarified.
- [Abstract, 'unlimited stream'] The phrase 'an unlimited stream of equally difficult, unseen instances' promises a property that cannot be established by finite sampling; the paper should explain how the difficulty preservation guarantee is obtained, rather than asserted.
Circularity Check
No significant circularity: the benchmark comparison and TFA metric are not constructed from their own outputs; the equal-difficulty assumption is a validity concern, not a circular step.
full rationale
The abstract reports a benchmark comparison between an Original Putnam set and a programmatically perturbed Variation set, with the central evidence being the drop in accuracy (o1-preview from 41.9% to 22.3%). There is no derivation chain in which a fitted parameter is later renamed as a prediction, no quantity is defined in terms of the target conclusion, and no load-bearing self-citation is invoked. The Variation protocol asserts, rather than proves, that perturbing variables and constants yields 'equally difficult' problems; this is an unvalidated empirical premise. If the premise fails, the memorization interpretation is weakened, but that is a correctness and external-validity concern, not circularity: the measured drop is not baked into the metric or into the construction of the Variation set. The TFA component is described as scoring reasoning traces, but the abstract provides no details indicating that it uses the evaluated model to grade itself or that the reported accuracy gaps are generated by TFA. Because no specific reduction from outputs back to inputs can be exhibited from the available text, no circular step meets the evidentiary bar, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (1)
- Perturbation protocol hyperparameters (variation count, substitution rules, difficulty matching thresholds) =
Not reported in abstract
assumptions (4)
- domain assumption Putnam problems in the Original set are substantially present in LLM training data, so performance on them can be inflated by memorization.
- ad hoc to paper Programmatic perturbation preserves problem difficulty while changing surface variables and constants.
- domain assumption Accuracy drop isolates memorization rather than format or notation distribution shift.
- domain assumption Teacher-Forced Accuracy (TFA) measures reasoning quality.
Cite this review
Pith. "Pith review of Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs." pith.science (2026). https://pith.science/paper/GLPKSP57
@misc{pith2026250808292,
author = {Pith},
title = {Pith review of: Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/GLPKSP57}},
note = {Machine review of arXiv:2508.08292}
}
read the original abstract
Current mathematical reasoning benchmarks for large language models (LLMs) are approaching saturation, with some achieving > 90% accuracy, and are increasingly compromised by training-set contamination. We introduce Putnam-AXIOM, a benchmark of 522 university-level competition problems drawn from the prestigious William Lowell Putnam Mathematical Competition, and Putnam-AXIOM Variation, an unseen companion set of 100 functional variants generated by programmatically perturbing variables and constants. The variation protocol produces an unlimited stream of equally difficult, unseen instances -- yielding a contamination-resilient test bed. On the Original set, OpenAI's o1-preview -- the strongest evaluated model -- scores 41.9%, but its accuracy drops by 19.6% (46.8% relative decrease) on the paired Variations. The remaining eighteen models show the same downward trend, ten of them with non-overlapping 95% confidence intervals. These gaps suggest memorization and highlight the necessity of dynamic benchmarks. We complement "boxed" accuracy with Teacher-Forced Accuracy (TFA), a lightweight metric that directly scores reasoning traces and automates natural language proof evaluations. Putnam-AXIOM therefore provides a rigorous, contamination-resilient evaluation framework for assessing advanced mathematical reasoning of LLMs. Data and evaluation code are publicly available at https://github.com/brando90/putnam-axiom.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.