Pith. sign in

REVIEW 3 major objections 5 minor 20 references

A 2,329-task execution benchmark over 5.52M hidden instances shows LLMs are brittle at exact combinatorial rule induction: best accuracy is ~56%, extra examples can wreck perfect solutions, and scaling adds ~10 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An execution-graded benchmark built from FindStat shows that current LLMs plateau near 50% on exact combinatorial code synthesis, with extra examples sometimes hurting and open-weight models matching closed ones.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A carefully built execution benchmark for exact code synthesis from mathematical descriptions, with real results about example sensitivity and reasoning-budget failures; the main caveat is that 'induction' vs 'memorization' is unresolved, but the paper is honest about it and worth refereeing. the 3 major comments →

arxiv 2607.18260 v1 pith:YKXUJOTD submitted 2026-05-16 cs.AI

FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis

Soham Dan This is my paper
classification cs.AI
keywords combinatorial code synthesisLLM evaluationprogram synthesisexecution-based benchmarkingfew-shot promptingsymbolic rule inductioncombinatorial statistics databasebijections
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a benchmark that turns 2,329 combinatorial definitions into closed-book programming tasks: given a mathematical description and up to five examples, a model must write one Python solve function that generalizes to hidden instances scored by exact execution. It claims this measures exact symbolic rule induction, a capability standard code benchmarks under-sample, and that on this axis current LLMs are brittle: the best open and closed models converge near 56% instance accuracy, statistics are easier than maps for every system, set partitions and binary trees are near zero, and long prompts cause a sharp accuracy cliff. It also documents two non-obvious regularities: five worked examples can collapse a perfect solution for a classical bijection to near zero, and five samples from a mid-tier model capture about the same task-accuracy headroom as an oracle over all eleven models. If the benchmark is valid, it is a reproducible instrument for a real capability gap rather than a ranking artifact, and its collection-level patterns separate structural difficulty from model-specific reasoning quality.

Core claim

The paper's central claim is that exact symbolic rule induction over structured combinatorial objects remains brittle across model scale, specialization, and provenance: the strongest open- and closed-source systems land within about one percentage point near 56% instance accuracy, an oracle over all eleven systems adds only about 10 points of task accuracy, and the same ceiling is reached by five-way sampling from one mid-tier model. The most specific discovery is that on several classical named bijections, five public examples actively regress a perfect zero-shot solution to near zero—failing even the public examples shown—because the model abandons a canonical implementation for a flawed

What carries the argument

The central object is the benchmark protocol itself: each of 2,329 tasks pairs a mathematical description with up to five public input-output examples and requires a single Python solve(obj), scored by exact sandboxed execution on millions of hidden combinatorial instances. The second load-bearing mechanism is the distinction between canonical-implementation mode and induce-from-examples mode, which explains why zero-example prompts can yield perfect classical bijections while five-example prompts collapse. The third is the pass@5 versus oracle saturation comparison, used to show that test-time scaling headroom is limited.

Load-bearing premise

The benchmark measures rule induction rather than retrieval of memorized content: the mathematical descriptions are public on the web, and the closed-book prompt strips identifiers and URLs but not the mathematical content itself.

What would settle it

Run the same closed-book protocol on a matched set of newly composed statistics—for example, sums or compositions of two published rules that could not have appeared in pretraining—and compare accuracy against the original 2,329 tasks. If accuracy on the novel tasks is far below the reported ~56% while the original tasks hold, the benchmark partly measures memorized retrieval rather than induction; if the two match, retrieval is not a major confound.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark measures what it claims, current LLMs are far from reliable at exact symbolic rule induction: best instance accuracy is about 56%, with near-zero performance on set partitions and binary trees.
  • Statistic synthesis (object to integer) is systematically easier than map synthesis (object to object) for all eleven systems, so map-to-object code synthesis is the harder frontier.
  • Few-shot examples are not monotonically helpful; on classical named bijections, five examples can destroy a perfect zero-shot solution, meaning example count and selection are critical prompt decisions.
  • Test-time scaling has a low ceiling: five samples from one mid-tier model reach roughly the same task-accuracy lift as an oracle over eleven models, suggesting limited diversity benefit.
  • Output-token budget must be reported and controlled when benchmarking reasoning models, because hidden reasoning can consume the visible response budget before any code is emitted.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If LLMs memorized the source database's descriptions and canonical implementations during pretraining—the zero-example conjugate-partition code is exactly the standard short implementation—then the absolute accuracy numbers measure retrieval as much as induction; the paper's open-book ablation only strips identifiers and URLs, not the mathematical content, so it cannot rule this out.
  • A natural extension is to synthesize new, never-published statistics by composing existing rules (e.g., sums or compositions of two published statistics) and test models on those; if accuracy drops sharply relative to the original tasks, memorization is a major driver.
  • The near-universal failure on set partitions and binary trees may reflect the linear string encoding of recursive objects rather than the underlying math; an encoding-ablation that changes serialization order could separate representation difficulty from induction difficulty.
  • The prompt-length cliff around 4–8k characters suggests instruction-following or context degradation is a separable bottleneck; chunking the description or restructuring the prompt could be a cheap test of that hypothesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FindStatBench, a benchmark of 2,329 combinatorial code-synthesis tasks (1,993 statistics, 336 maps) derived from FindStat, with 5.52M hidden instances scored by exact sandboxed execution. For each task a model receives a mathematical description and at most five public examples and must emit a single Python solve function with no retrieval, tools, or execution feedback. Eleven systems are evaluated (four closed-source, seven open-weight). Headline findings are: the strongest open- and closed-source models converge within ~1 pp instance accuracy; an 11-model oracle improves task accuracy by only ~10 pp, and pass@5 on one mid-tier model gives a similar lift; additional examples can sharply hurt performance on several classical bijections; output-token budget is a first-class failure mode for reasoning models; statistic synthesis is universally easier than map synthesis; set partitions and binary trees are near-zero; long prompts induce a sharp accuracy cliff. The paper ships a reproducibility package including crawler, evaluator, prompt exports, split manifests, and per-model submissions.

Significance. If the validity concerns below are addressed, FindStatBench is a useful instrument for an under-measured capability: writing short, exact programs from formal combinatorial descriptions. The benchmark engineering is careful and transparent: exact execution in a sandbox, AST anti-cheating checks with no evidence of large-literal or example-branching artifacts, public/hidden splits, per-model failure decomposition, cost-normalized comparisons, and a dedicated limitations section. The dataset-level patterns — statistics easier than maps, near-zero collections, the prompt-length cliff, and the non-monotone effect of examples — are genuine, falsifiable observations about current LLMs. The paper is also honest about its contamination-control limits, which is a strength; the issue is that the paper's central 'induction' framing requires stronger evidence than the current ablation provides.

major comments (3)
  1. [§2, §7, App. L, Fig. 3] The central claim that FindStatBench measures exact symbolic rule induction (Abstract; §4–6) is not fully separated from memorized retrieval. The corpus is entirely public FindStat content, and §2 concedes "the underlying mathematical descriptions remain public on the web." The open/closed-book ablation (App. L) varies only identifiers, URLs, and source branding, so a <1 pp gap does not show that models have not memorized the mathematical content or canonical implementations. Fig. 3 provides direct evidence of retrieval: the zero-shot Mp00044 solution is exactly the canonical conjugate-partition implementation. This affects three headline findings: (i) absolute accuracies are upper bounds on induction; (ii) Table 5's "examples can hurt" may instead be "five examples overwrite a memorized canonical algorithm"; (iii) the near-zero collections may reflect absence of memorized content rather
  2. [§3, §5, App. B] The gpt-oss models are scored from their hidden reasoning trace when the visible content is empty (24% and 35% of responses at max_tokens=4,000), while other models are scored only on visible content, except for GPT-5.4-mini's separately documented retry. This asymmetry is material to the headline open/closed convergence claim: gpt-oss-120b's 55.13% instance accuracy includes code that never appeared in the visible answer. Please report a visible-only variant for the gpt-oss line and quantify how many perfect tasks or correct hidden instances depend on trace extraction. Without this, the "same protocol" framing and the ~1 pp convergence are not strictly established.
  3. [§6.3, Fig. 4] The pass@5 experiment is run on a 497-task subset of one model (Qwen3-235B), while the 11-model oracle is computed on the full 2,329-task corpus. The claim that five-way sampling "reaches the same ceiling" therefore compares a subset lift (+9.90 pp) with a full-corpus lift (+11.25 pp). Because the subset may have different headroom, the two curves in Fig. 4 are not directly comparable. Please compute pass@5 on the full corpus (or the ensemble lift on the same 497-task subset) before concluding that both saturate at the same ~10 pp ceiling.
minor comments (5)
  1. [§4 / Table 7] The text reports Qwen3-235B-A22B-Instruct reaches "996 perfect tasks per dollar," but Table 7 lists 1000.0 perfect per dollar. Please reconcile.
  2. [§6.4] The seed-variance estimate (std 0.82 pp task, 1.29 pp instance) is for Qwen3-235B-A22B-Instruct at temperature 0.7 only. The main runs for other models are greedy or low-temperature. The statement that the 0.92 pp gap between gpt-oss-120b and Claude Sonnet 4.6 is "inside the noise floor" should be restricted to this model/temperature, or accompanied by variance estimates for the two models being compared.
  3. [Table 2 / App. C] The GPT-5.4-mini row is the merged run with a 6,400-token retry; App. C shows 95.53% generation rate, while the raw run has substantially lower coverage (Fig. 5). Please mark this in the table caption or footnote so readers do not misread the row as a single-pass run.
  4. [App. L] The appendix heading "Self-repair on the open-weight frontier" also contains the open-book vs. closed-book ablation and the reasoning-budget scaling run. Consider splitting or renaming for navigability.
  5. [§2] The prompt-support note says 2,325 of 2,329 tasks expose five examples. Since the example-count ablation (Table 16) is on a 199-task subset, consider stating explicitly in §6.2 that the n=0/n=1/n=3/n=5 results are on that subset rather than the full corpus, to avoid an apparent inconsistency with the full-corpus prompt description.

Circularity Check

0 steps flagged

No circular derivation: FindStatBench is an external evaluation benchmark; results are produced by executing external models against independent FindStat data, with no fitted-parameter-then-prediction structure and no load-bearing self-citation.

full rationale

The paper does not contain a derivation chain in which an output is equivalent to an input. FindStatBench tasks and hidden instances are constructed from FindStat's public pages and scored by exact sandboxed execution of model-generated Python code (Abstract; Sections 2-3). Model accuracies are measured results, not predictions derived from fitted parameters. The paper explicitly reports an open-book vs stripped-identifier ablation (App. L) and concedes that mathematical descriptions remain public (Section 7), but this is a contamination/construct-validity caveat rather than circularity: no quantity is fit and then relabeled a prediction, and no benchmark score is defined in terms of the models' outputs. The only external citation to the source database, Rubey and Stump, 2019, is used for data provenance, not to justify the benchmark's conclusions. No uniqueness theorem, ansatz, or self-citation is invoked as load-bearing. Thus the benchmark claims rest on external execution evidence, and any concerns about memorization affect interpretation of what the scores measure, not circularity of the method.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

This is an empirical evaluation paper, so the ledger holds protocol thresholds rather than fitted model parameters. The four free parameters are hand-chosen evaluation settings that the headline numbers are sensitive to — most notably the 5-example cap (which the paper itself shows is non-monotone in performance) and the output-token budgets (identified as a first-class hyperparameter). The axioms are the standard validity conditions of benchmarking — prompt unambiguity, correct output comparison, cross-provider comparability, transferable noise estimates — all disclosed in Section 7/appendix. No invented entities: the benchmark is a dataset/artifact, not a postulated entity.

free parameters (4)
  • public-example cap and fraction = 5 examples; 30% of visible pairs
    Hand-chosen split parameters (App. A). The cap yields nearly degenerate prompt support (2,325/2,329 tasks with exactly 5 examples) and directly conditions the "examples can hurt" finding (§6.2): the non-monotone n∈{0,1,3,5} curve means measured performance depends on this threshold.
  • output-token budgets = max_tokens=1,600 (default), 4,000 (gpt-oss line), 6,400 (GPT-5.4-mini retry), 8,000 (ablation)
    Section 5's "reasoning ate the answer" finding is defined against these caps: 36%/46% empty responses at 1,600 tokens; raising to 8,000 rescues +3.46 pp (App. L). The budget is a first-class hyperparameter by the paper's own conclusion.
  • sandbox limits = 2 s timeout; 256 MB memory
    Hand-set evaluator constraints (App. A); a correct-but-slow or memory-hungry solution is scored as a failure on tasks with large combinatorial objects.
  • evaluation subsets = 199-task example-ablation subset; 497- and 500-task variance subsets
    Chosen by hand (§6.2, §6.4, App. K); the small subsets support the pass@5-vs-oracle and example-count headroom claims, so their composition is load-bearing for those numbers.
axioms (4)
  • domain assumption FindStat textual descriptions plus ≤5 examples uniquely determine the intended statistic or map among all possible programs.
    The benchmark scores any implementation that matches hidden instances; if descriptions are ambiguous, low accuracy conflates ambiguity with model failure (§2; prompt template App. O).
  • domain assumption Exact match after canonicalisation correctly judges map equality in the sandbox.
    The paper's own Limitations (§7) admit "a small fraction of equivalent objects in non-canonical encodings can be undercounted," which would depress map-synthesis accuracy for every model.
  • domain assumption Single-window runs of API-served models with provider-specific knobs are comparable across systems.
    Limitations (§7) state reasoning-effort, output-token, and response-mode controls differ per provider and Together AI's hardware/quantisation is undisclosed; the main comparisons treat the standardised settings as fair baselines (App. B).
  • domain assumption The variance measured on one model at T=0.7 bounds run-to-run variability of other models at greedy decoding.
    Section 6.4 uses Qwen3-235B std <1.3 pp to call the 0.92 pp gap between gpt-oss-120b and Claude Sonnet 4.6 "inside the noise floor," extending a within-model, non-greedy estimate to different models and settings.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis." pith.science (2026). https://pith.science/paper/YKXUJOTD

@misc{pith2026260718260,
  author       = {Pith},
  title        = {Pith review of: FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKXUJOTD}},
  note         = {Machine review of arXiv:2607.18260}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statistic synthesis, which maps objects to integers, and map synthesis, which maps objects to objects. Each task gives a mathematical description and at most five public input-output examples; a model must emit one Python solve function with no retrieval, tool use, execution feedback, voting, or reranking. Submissions are scored by exact sandboxed execution on held-out combinatorial objects. We evaluate eleven systems: four closed-source production models and seven open-weight models served through one inference provider. FindStatBench reveals three main patterns. First, the strongest open- and closed-source systems converge within 1 pp instance accuracy, and both an oracle over all systems and five-way sampling from one mid-tier model yield only limited task-accuracy gains. Second, examples can hurt: several classical bijections are solved perfectly with zero examples but fail under five-example prompts. Third, some failures reflect output-budget mechanics, as reasoning can exhaust the visible response before code is emitted. Overall, statistic synthesis is much easier than map synthesis, some collections remain near-zero, long prompts cause a sharp accuracy cliff, and exact symbolic rule induction remains brittle.

Figures

Figures reproduced from arXiv: 2607.18260 by Soham Dan.

Figure 1
Figure 1. Figure 1: FindStatBench at a glance. Curated FindStat pages become closed-book prompts that [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cost vs. instance accuracy for all eleven systems; circles closed-source, triangles open [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qwen3-235B-A22B-Instruct code for Mp00044 (conjugate partition; Integer partitions [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Two paths to a ∼10 pp lift on FindStatBench. Red triangles: pass@k on five samples of Qwen3-235B-A22B-Instruct at temperature 0.7. Blue circles: cumulative ensemble@k task accuracy as we add models in best-first order to the closed/open joint pool. Both curves saturate near ∼10 pp lift over k=1—a single mid-tier model with 5-way sampling extracts headroom of the same order as an 11-model multi-provider ens… view at source ↗
Figure 5
Figure 5. Figure 5: Initial generation rate versus covered-instance accuracy before the 6,400-token GPT-5.4- [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Model × collection heatmap for the eight representative collections reported for all eleven systems. Color encodes strict instance accuracy (%): plane partitions, binary words, and integer partitions are broadly easier, while set partitions and binary trees remain near zero. Collection Tasks GPT-5.4-mini Claude Haiku 4.5 Claude Sonnet 4.6 Gemini 2.5 Flash Graphs 332 61.1 51.0 48.9 16.6 Posets 96 47.9 35.9 … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 9 linked inside Pith

  1. [1]

    Rubey, Martin and Stump, Christian , booktitle =

  2. [2]

    Notices of the American Mathematical Society , year =

    Fingerprint Databases for Theorems , author =. Notices of the American Mathematical Society , year =

  3. [3]

    Mathematics in Computer Science , year =

    Ber. Mathematics in Computer Science , year =

  4. [4]

    Foundations and Trends in Programming Languages , year =

    Program Synthesis , author =. Foundations and Trends in Programming Languages , year =

  5. [5]

    2021 , eprint =

    Evaluating Large Language Models Trained on Code , author =. 2021 , eprint =

    2021

  6. [6]

    2021 , eprint =

    Program Synthesis with Large Language Models , author =. 2021 , eprint =

    2021

  7. [7]

    Measuring Coding Challenge Competence With

    Hendrycks, Dan and Basart, Steven and Kadavath, Saurav and Mazeika, Mantas and Arora, Akul and Guo, Ethan and Burns, Collin and Puranik, Samir and He, Horace and Song, Dawn and Steinhardt, Jacob , year =. Measuring Coding Challenge Competence With. 2105.09938 , archivePrefix =

    Pith/arXiv arXiv

  8. [8]

    and Yu, Tao , booktitle =

    Lai, Yuhang and Li, Chengxi and Wang, Yiming and Zhang, Tianyi and Zhong, Ruiqi and Zettlemoyer, Luke and Yih, Scott Wen-tau and Fried, Daniel and Wang, Sida I. and Yu, Tao , booktitle =. 2023 , pages =

    2023

  9. [9]

    2406.15877 , archivePrefix =

    Zhuo, Terry Yue and Vu, Minh Chien and Chim, Jenny and Hu, Han and Yu, Wenhao and Widyasari, Ratnadira and Yusuf, Imam Nur Bani and Zhan, Haolan and He, Junda and Paul, Indraneil and others , year =. 2406.15877 , archivePrefix =

    Pith/arXiv arXiv

  10. [10]

    Competition-level code generation with

    Li, Yujia and Choi, David and Chung, Junyoung and Kushman, Nate and Schrittwieser, Julian and Leblond, R. Competition-level code generation with. Science , year =

  11. [11]

    2403.07974 , archivePrefix =

    Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , year =. 2403.07974 , archivePrefix =

    Pith/arXiv arXiv

  12. [12]

    2102.04664 , archivePrefix =

    Lu, Shuai and Guo, Daya and Ren, Shuo and Huang, Junjie and Svyatkovskiy, Alexey and Blanco, Ambrosio and Clement, Colin and Drain, Dawn and Jiang, Daxin and Tang, Duyu and Li, Ge and Zhou, Lidong and Shou, Linjun and Zhou, Long and Tufano, Michele and Gong, Ming and Zhou, Ming and Duan, Nan and Sundaresan, Neel and Deng, Shao Kun and Fu, Shengyu and Liu,...

    Pith/arXiv arXiv

  13. [13]

    and Guha, Arjun and Greenberg, Michael and Jangda, Abhinav , year =

    Cassano, Federico and Gouwar, John and Nguyen, Daniel and Nguyen, Sydney and Phipps-Costin, Luna and Pinckney, Donald and Yee, Ming-Ho and Zi, Yangtian and Anderson, Carolyn Jane and Feldman, Molly Q. and Guha, Arjun and Greenberg, Michael and Jangda, Abhinav , year =. 2208.08227 , archivePrefix =

    Pith/arXiv arXiv

  14. [14]

    Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , year =

    Learning to Map Natural Language to Code in a Data-Sparse Setting , author =. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics , year =

  15. [15]

    2019 , volume =

    Kulal, Sumith and Pasupat, Panupong and Chandra, Kartik and Lee, Mina and Padon, Oded and Aiken, Alex and Liang, Percy , booktitle =. 2019 , volume =

    2019

  16. [16]

    Measuring Mathematical Problem Solving With the

    Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob , year =. Measuring Mathematical Problem Solving With the. 2103.03874 , archivePrefix =

    Pith/arXiv arXiv

  17. [17]

    2109.00110 , archivePrefix =

    Zheng, Kunhao and Han, Jesse Michael and Polu, Stanislas , year =. 2109.00110 , archivePrefix =

    Pith/arXiv arXiv

  18. [18]

    2409.12186 , archivePrefix =

    Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Lu, Keming and Dang, Kai and Yang, Yang and Fan, Rui and Su, Junyang and Wang, Peng and Chen, Limin and Li, Minrui and Lin, Bingquan and Xie, Tao and Ge, Wei and others , year =. 2409.12186 , archivePrefix =

    Pith/arXiv arXiv

  19. [19]

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Anirudh and Schelten, Alan and Yang, Amy and Fan, Angela and others , year =. The. 2407.21783 , archivePrefix =

    Pith/arXiv arXiv

  20. [20]

    2025 , howpublished =

    2025

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.