Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A code-generating automated protocol, CPro1, guided by a reasoning large language model, constructs verified solutions to long-open combinatorial design problems, resolving open cases for 7 of 16 problem types from the 2006 Handbook of…

desk verdict Verified constructions, not the LLM protocol, are what make this paper worth a referee's time. read the letter →

arxiv 2505.23881 v1 pith:LHK67XGO submitted 2025-05-29 cs.AI cs.CLmath.CO

Christopher D. Rosin This is my paper
classification cs.AIcs.CLmath.CO MSC 05B30
keywords combinatorialdesignopeninstanceslargelanguagemodelsreasoningcodegenerationsearchheuristicsexistenceproblemsverifiedconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that an automated protocol, CPro1, can use a reasoning large language model to write and tune search programs that construct explicit solutions to long-open combinatorial design existence problems. The authors report verified solutions for 7 of 16 problem types selected from the 2006 Handbook of Combinatorial Designs, including cases in Bhaskar Rao Designs, Symmetric Weighing Matrices, and Balanced Ternary Designs that CPro1 with non-reasoning models had not solved. They also report new Covering Sequences, Johnson Clique Covers, Deletion Codes, and a Uniform Nested Steiner Quadruple System from recent literature. If the results hold, the listed instances are no longer open and the approach offers a way to attack small existence questions without writing bespoke mathematics proofs.

What carries the argument

The load-bearing object is the Constructive Protocol CPro1: an automated experiment loop that turns a textual design definition and a Python validity verifier into candidate search programs. It generates 1,000 candidate C implementations from LLM proposals, tunes each candidate's hyperparameters on development instances using the verifier as a score function, keeps the top five for code optimization, selects the top two after two-hour runs, and finally runs those for 48 hours on open instances, outputting only results that pass the verifier. The reasoning LLM supplies the heuristic ideas -- simulated annealing, tabu search, depth-first search, genetic algorithms, and two-phase decompositions -- while the verifier supplies correctness.

What would settle it

Recompute each claimed solution independently: parse the arrays and code words from the paper's published outputs and check them against the design definitions without relying on the supplied verifier, then audit each instance's status against current authoritative tables. If any array fails the definitional constraints or an instance turns out to have been solved before, the central claim has an error.

Watch

Extended reading notes

Core claim

The central discovery is that a protocol which repeatedly asks a reasoning LLM to propose diverse search strategies in C, then tunes hyperparameters, scores candidates against known development instances, optimizes the fastest candidates, and lets survivors run for 48 hours on open instances, can produce verified combinatorial designs where existence was previously unknown. The paper's specific claim is that CPro1 with the o3-mini-high reasoning model resolves open instances for 7 of the 16 Handbook problems, and that for three of those -- Bhaskar Rao Designs, Symmetric Weighing Matrices, and Balanced Ternary Designs -- the reasoning model succeeds where the same protocol with a non-reasoning model did not. The outputs are explicit arrays, matrices, or code words that pass a Python verifier implementing the design definition.

Load-bearing premise

The load-bearing premise is that the Python verifier encodes each design definition correctly and that the cited sources' open-instance lists are complete; if either fails, a listed 'solved' instance may not actually be solved or may not have been open.

Editorial extensions

If this is right

  • For every solved instance listed in the paper's results, an explicit certified construction now exists, so the existence question for those parameter values is settled.
  • The three newly solved Handbook classes (Bhaskar Rao Designs, Symmetric Weighing Matrices, Balanced Ternary Designs) gain at least one new instance each, shrinking the open-instance lists for those problems.
  • The Deletion Code results improve on the previously reported LLM-search state of the art for small parameters, providing larger sets of codewords than earlier methods.
  • The protocol provides a template for attacking other small open existence questions: supply a definition, a verifier, and development instances, and let the protocol search for a heuristic.
  • For problem types where the community has already invested heavily in specialized search, such as Covering Arrays, the protocol does not necessarily compete; its successes come from less-explored design types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: If the verifier-based protocol can be trusted, the bottleneck for automating combinatorial existence shifts from the search itself to producing correct verifiers and accurate open-instance lists; one could test this by applying the same pipeline to a fresh blind set of open instances.
  • Extension: The two-phase strategies that emerged (build the 0/1 incidence pattern before assigning signs, or build a Steiner system before nesting pairs) suggest that reasoning models can rediscover decomposition principles that are not explicitly prompted, which would be worth studying as a general pattern in LLM-guided search.
  • Extension: The paper's single full run per problem leaves stochasticity untested; a natural next experiment would be repeated runs on the same problems to measure run-to-run success rates and whether the solved instances are stable.
  • Extension: Because the successful programs are randomized heuristics with little visible structure, the method proves existence but does not automatically yield human-readable mathematical constructions; an independent mathematical verification of the published arrays could turn these computational discoveries into theorems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper applies the Constructive Protocol CPro1, an LLM-based pipeline that generates and tunes search heuristics in C, to open existence instances of combinatorial designs. Using the reasoning model o3-mini-high, it reports solving open instances for 7 of 16 design types selected from the 2006 Handbook of Combinatorial Designs, including three types (Bhaskar Rao Designs, Symmetric Weighing Matrices, Balanced Ternary Designs) that were not solved by an earlier CPro1 run with a non-reasoning model. It also reports new Covering Sequences, Johnson Clique Covers, Uniform Nested Steiner Quadruple Systems, and improved Deletion Codes relative to a recent FunSearch-based result. The existence claims are supported by explicit constructions listed in the appendix and repository, each checked by a Python verifier rather than by the LLM's own assertion. The paper includes ablations, scaled-down repeated runs, and a limitations section that discloses single-run, nondeterministic behavior and the non-existence-proof limitation.

Significance. If the listed constructions and their verifiers are correct, the paper provides a notable positive demonstration that LLM-generated heuristics can resolve open existence questions in combinatorics, with independently checkable artifacts. The contributions include concrete constructions for several previously open parameter sets, a clear protocol, ablations showing the role of each pipeline component, and repeated scaled-down runs that give some evidence against pure luck. The main risk is not circularity: the load-bearing outputs are explicit arrays/sequences/codes checked by an external verifier. The significance is somewhat tempered by the paper's own disclosure that these are single full-scale runs and that some successes come from the prototyping set used to develop the method.

major comments (4)
  1. [Appendix Tables 5–8; Algorithm 1] The central load-bearing artifact is the Python verifier, but the manuscript does not reproduce it or provide a formal mapping from each verifier check to the cited mathematical definition; the appendix only gives prompt definitions. Because Algorithm 1 certifies outputs solely through this verifier, a definitional mismatch would invalidate the claimed existence of the listed designs. Please include the verifier source for each design type and state explicitly which definitional convention is used.
  2. [Table 5 (BTD definition)] The appendix defines pair occurrence in a Balanced Ternary Design as sum_b m_{vb} m_{wb} = L, so a block containing v twice and w once contributes 2 to the pair count. If the Handbook or the open-instance literature counts each block at most once per pair, then the certified (16,22;9,1,11;8,5) array would not be the object whose existence was open. Please state the convention for BTD pair occurrence and confirm that the verifier matches the literature.
  3. [Table 7 (Covering Sequence; Johnson Clique Cover)] Two other definitional conventions need confirmation. The Covering Sequence definition in Table 7 says 'length at most L', but Table 1(b) lists L as the solved parameter; if the open instances in [5] are stated for exact length L, a shorter sequence does not resolve them. For Johnson Clique Cover, the appendix defines a cover of all vertices, while some graph-theoretic sources define a clique cover of all edges; the manuscript should confirm which notion is used in [25] and that the verifier matches it.
  4. [Section 3.1; Table 1] The claim that 7 of 16 Handbook problems are 'newly solved' depends on the completeness of the open-instance lists taken from [7] and the updates [8, 13, 16, 18], plus the hand-filtered February 2025 scan. The manuscript does not audit these lists per problem. A missed prior construction would not affect the existence of the posted design, but it would affect the novelty claim; please supply an open-status audit or cite authoritative updated tables for each problem.
minor comments (5)
  1. [Section 4.2, Figure 1] The caption reports p<0.0001 from a Z-test on 25 scaled-down runs; please state whether the four comparisons are adjusted for multiple testing and clarify the null hypothesis.
  2. [Table 7 versus Table 1(b)] The Covering Sequence appendix states n<=16 and L<=1200, but Table 1(b) includes n up to 20 and L up to 207000; indicate whether these are the 'adjacent instance' extensions described in Section 4 and correct the stated range.
  3. [Table 8] In the Deletion Code definition, 'deleting two distinct bits' should read 'deleting s distinct bits'.
  4. [Table 3 caption] The caption says 'for each LLM and total number of candidate programs' but no candidate-count column is shown; please correct the caption.
  5. [References] The open-status updates [8, 13, 16, 18] are cited as URLs and dates; provide version or access-date information so the open-status audit is reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: existence claims are explicit constructions verified against independent definitional constraints; self-citation to [40] is not load-bearing.

full rationale

The paper's central claims are existence results for specific combinatorial designs (e.g., SymmW(22,16), BRD(15,42,14,5,4), BTD(16,22;9,1,11;8,5)), supported by explicit constructions posted in the repository and by sample solutions in Appendix B, each checked by a Python verifier that encodes the defining constraints. The verifier is an input to CPro1, not an output of the LLM, and the same verifier is used to score development-instance candidates and to certify final open-instance constructions; no parameter is fitted to the open instances and then renamed as a prediction. Hyperparameters are tuned on separate development instances, and open instances are run only after candidate selection, so the successes are not fitted values by construction. The prototyping-set instances are marked with a dagger and are independently verified; using them as results is a selection-effect caveat, not a definitional reduction. The only self-citation is to [40], the author's earlier protocol paper; Algorithm 1 restates the protocol here, and the GPT-4o comparison is a baseline, not a load-bearing step for the existence of the posted designs. The Limitations section discloses nondeterminism and single runs, and the skeptic's concerns about verifier fidelity or open-status completeness are correctness/novelty risks, not circularity. No equation or construction in the paper reduces to its input by definition.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim is empirical: LLM-generated heuristics produced verified constructions. No free parameter is fitted into the existential claim itself; the protocol constants above are hand-set engineering choices. The main load-bearing assumptions are the correctness of the Python verifiers and the accuracy of the open-instance literature. No new mathematical entities are postulated.

free parameters (3)
  • candidate program count (1000 = 50 reps x 20 strategies) = 1000
    Chosen by hand for CPro1; affects coverage of the heuristic space. Ablations show removing components changes results, but this count is a protocol constant, not a mathematical parameter.
  • execution budgets (50s scoring, 2h dev, 48h open) = 50s / 2h / 48h
    Hand-set runtime budgets in Algorithm 1. They determine which heuristics survive truncation, so they influence which instances get solved, but the final constructions are verified independently.
  • hyperparameter tuning grid budgets = 1000 pts x 0.5s, top 100 x 5s, top 10 x 50s
    Hand-set grid schedule for per-candidate hyperparameter tuning. This is fitting of search heuristics, but existence claims do not depend on tuned values.
assumptions (3)
  • domain assumption Python verifiers correctly encode the definitions of each combinatorial design
    The protocol uses the verifier to score candidates and to certify final solutions (Algorithm 1, Section 3.1). If a verifier has a bug, a 'verified solution' may be invalid. The verifiers are provided only via the GitHub repository, not in the manuscript.
  • domain assumption Open-instance lists from the 2006 Handbook and 2025 articles are accurate
    The paper omits instances it believes were solved in [43,35,17,13,8,16,18,9]. The 'newly solved' classification depends on the completeness and correctness of these literature lists.
  • standard math The theorem that JCC clique covers need only maximal cliques of type 0 or 1
    Given as a prompt definition in the appendix (Table 7) and used to restrict the search space for Johnson Clique Cover. It is attributed to the literature and not proved in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problems." pith.science (2026). https://pith.science/paper/LHK67XGO

@misc{pith2026250523881,
  author       = {Pith},
  title        = {Pith review of: Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHK67XGO}},
  note         = {Machine review of arXiv:2505.23881}
}
read the original abstract

Large Language Models (LLMs) with reasoning are trained to iteratively generate and refine their answers before finalizing them, which can help with applications to mathematics and code generation. We apply code generation with reasoning LLMs to a specific task in the mathematical field of combinatorial design. This field studies diverse types of combinatorial designs, many of which have lists of open instances for which existence has not yet been determined. The Constructive Protocol CPro1 uses LLMs to generate search heuristics that have the potential to construct solutions to small open instances. Starting with a textual definition and a validity verifier for a particular type of design, CPro1 guides LLMs to select and implement strategies, while providing automated hyperparameter tuning and execution feedback. CPro1 with reasoning LLMs successfully solves long-standing open instances for 7 of 16 combinatorial design problems selected from the 2006 Handbook of Combinatorial Designs, including new solved instances for 3 of these (Bhaskar Rao Designs, Symmetric Weighing Matrices, Balanced Ternary Designs) that were unsolved by CPro1 with non-reasoning LLMs. It also solves open instances for several problems from recent (2025) literature, generating new Covering Sequences, Johnson Clique Covers, Deletion Codes, and a Uniform Nested Steiner Quadruple System.

Figures

Figures reproduced from arXiv: 2505.23881 by the authors.

Figure 1
Figure 1. Results from scaled￾down 40-candidate runs. With re￾duced runtime (2 hours) on open instances, no final dev test, and no optimization. Each bar shows the rate of successfully solving at least one open instance, across 25 repeated runs, with 95% Clopper￾Pearson confidence interval. Each problem’s difference between o3- mini-high and GPT-4o is significant (p<0.0001, Z-test). 1000/40 = 25 scaled-down runs. Each bar sho… view at source ↗
Figure 2
Figure 2. Symmetric Weighing Matrix with n=22 w=16 [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Bhaskar Rao Design with parameters (15,42,14,5,4). Note the transpose is shown. [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Balanced Ternary Design with parameters (16,22;9,1,11;8,5) [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]
Figure 5
Figure 5. Figure 5: Covering Sequence with n=9 R=1 L=71 19 [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Johnson Clique Cover with N=13 k=4 C=105 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Uniform Nested Steiner Quadruple System with v=14 p=91 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Deletion Code with n=12 s=2 m=36 22 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Improved lower bounds for the Shannon capacity of odd cycles

    cs.IT 2026-07 conditional novelty 6.0 of 10

    New independent sets improve Shannon-capacity lower bounds to Θ(C7)>3.258020, Θ(C11)>5.289773, Θ(C13)>6.300109, Θ(C15)>7.301399.

  2. Covering Sequences and Covering-Sequences Codes

    cs.IT 2026-07 conditional novelty 6.0 of 10

    Hamming codes, read as constacyclic codes, yield covering sequences whose length is within a factor q/(q−1) of optimal for every prime power q.

  3. Improved Upper Bounds for Slicing the Hypercube

    cs.AI 2026-02 conditional novelty 6.0 of 10

    All edges of the n-dimensional hypercube can be sliced with at most 4n/5 hyperplanes (with a small odd-multiple-of-5 exception), improving the 1971 Paterson bound of 5n/6 via an explicit 8-hyperplane slicing of Q10.

Reference graph

Works this paper leans on

56 extracted references · 38 canonical work pages · cited by 3 Pith papers

  1. [5]

    Y . M. Chee, T. Etzion, H. Ta, and V . K. Vu. Constructions of covering sequences and arrays,

    work page

  2. [25]

    S. F. Jørgensen. On the clique covering numbers of Johnson graphs, 2025. arXiv:2502.15019

    work page Pith review arXiv 2025

  3. [7]

    C. J. Colbourn and J. H. Dinitz.Handbook of combinatorial designs. Taylor & Francis, 2006

    work page 2006

  4. [1]

    Bömer, N

    T. Bömer, N. Koltermann, M. Disselnmeyer, L. Dörr, and A. Meyer. Leveraging Large Language Models to develop heuristics for emerging optimization problems, 2025. arXiv:2503.03350

    arXiv 2025

  5. [2]

    Charton, J

    F. Charton, J. S. Ellenberg, A. Z. Wagner, and G. Williamson. PatternBoost: Constructions in mathematics with a little help from AI, 2024. arXiv:2411.00566

    arXiv 2024

  6. [3]

    H. Chau, H. Jenne, D. Brown, J. He, M. Raugas, S. Billey, and H. Kvinge. Machine learning meets algebraic combinatorics: A suite of datasets capturing research-level conjecturing ability in pure mathematics, 2025. arXiv:2503.06366

    work page Pith review arXiv 2025

  7. [4]

    Y . M. Chee, S. H. Dau, T. Etzion, H. M. Kiah, and W. Zhang. Pairs in nested Steiner quadruple systems.Journal of Combinatorial Designs, 33(5):177–187, 2025

    work page 2025

  8. [6]

    Z. Chen, Z. Zhou, Y . Lu, R. Xu, L. Pan, and Z. Lan. UBER: Uncertainty-based evolution with large language models for automatic heuristic design, 2024. arXiv:2412.20694

    arXiv 2024

Show all 56 references
  1. [8]

    J. Dinitz. New results in Part V, 2018. https://site.uvm.edu/jdinitz/?page_id=404

    2018

  2. [9]

    J. Dinitz. New results in Part VI, 2018. https://site.uvm.edu/jdinitz/?page_id=413

    2018

  3. [10]

    J. S. Ellenberg, C. S. Fraser-Taliente, T. R. Harvey, K. Srivastava, and A. V . Sutherland. Generative modeling for mathematical discovery, 2025. arXiv:2503.11061

    2025 arXiv

  4. [11]

    AI benchmarking hub – FrontierMath, 2025

    EpochAI. AI benchmarking hub – FrontierMath, 2025. https://epoch.ai/data/ai-benchmarking- dashboard

    2025

  5. [12]

    T. A. Feo and M. G. Resende. Greedy randomized adaptive search procedures.Journal of global optimization, 6:109–133, 1995

    1995

  6. [13]

    S. D. Georgiou, S. Stylianou, and H. Alrweili. On symmetric weighing matrices.Mathematics, 11:2076, 2023

    2023

  7. [14]

    Glazer, E

    E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. d. O. Santos, et al. FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI, 2024. arXiv:2411.04872

    2024 arXiv

  8. [15]

    Glover and M

    F. Glover and M. Laguna.Tabu search. Springer, 1997

    1997

  9. [16]

    D. Gordon. La Jolla Covering Repository Tables, 2025. https://ljcr.dmgordon.org/cover/table.html

    2025

  10. [17]

    M. Greig. Constructions using balanced n-ary designs. InDesigns, pages 227–274. Springer, 2002

    2002

  11. [18]

    T. S. Griggs and A. R. Kozlik. The last two perfect Mendelsohn designs with block size 5. Journal of Combinatorial Designs, 28(12):865–868, 2020

    2020

  12. [19]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. arXiv:2501.12948

    2025 arXiv

  13. [20]

    C. He, R. Luo, Y . Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y . Huang, Y . Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. OlympiadBench: A challenging benchmark for promoting AGI with Olympiad-level bilingual multimodal scientific problems, 2024. arXiv:2402.14008

    2024 arXiv

  14. [21]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the MATH dataset.NeurIPS, 2021. 9

    2021

  15. [22]

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang. Large language models for software engineering: A systematic literature review.ACM Trans. Softw. Eng. Methodol., 33(8), 2024. doi: 10.1145/3695988

    2024 doi

  16. [23]

    Hurst, A

    A. Hurst, A. Lerer, et al. GPT-4o system card, 2024. arXiv:2410.21276

    2024 arXiv

  17. [24]

    Jiang, F

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim. A survey on large language models for code generation, 2024. arXiv:2406.00515

    2024 arXiv

  18. [26]

    Kirkpatrick, C

    S. Kirkpatrick, C. D. Gelatt Jr, and M. P. Vecchi. Optimization by simulated annealing.Science, 220(4598):671–680, 1983

    1983

  19. [27]

    D. E. Knuth. Dancing links, 2000. arXiv:cs/0011047

    2000 arXiv

  20. [28]

    F. Liu, X. Tong, M. Yuan, X. Lin, F. Luo, Z. Wang, Z. Lu, and Q. Zhang. Evolution of heuristics: towards efficient automatic algorithm design using large language model. InProceedings of the 41st International Conference on Machine Learning, ICML’24, 2024

    2024

  21. [29]

    J. Liu, X. Lin, J. Bayer, Y . Dillies, W. Jiang, X. Liang, R. Soletskyi, H. Wang, Y . Xie, B. Xiong, et al. Generating streamlining constraints with Large Language Models, 2024. arXiv:2408.10268

    2024

  22. [30]

    J. Liu, X. Lin, J. Bayer, Y . Dillies, W. Jiang, X. Liang, R. Soletskyi, H. Wang, Y . Xie, B. Xiong, et al. CombiBench: Benchmarking LLM capability for combinatorial mathematics, 2025. arXiv:2505.03171

    2025 arXiv

  23. [31]

    LiveBench leaderboard, 2025

    LiveBench. LiveBench leaderboard, 2025. https://livebench.ai

    2025

  24. [32]

    Mehrabian, A

    A. Mehrabian, A. Anand, H. Kim, N. Sonnerat, M. Balog, G. Comanici, T. Berariu, A. Lee, A. Ruoss, A. Bulanova, et al. Finding increasingly large extremal graphs with AlphaZero and tabu search. InNeurIPS, 2024

    2024

  25. [33]

    Mitchell.An introduction to genetic algorithms

    M. Mitchell.An introduction to genetic algorithms. MIT Press, 1998

    1998

  26. [34]

    Nejjar, L

    M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber. LLMs for science: Usage for code generation and data analysis.Journal of Software: Evolution and Process, 37(1):e2723, 2025

    2025

  27. [35]

    Noritake, M

    H. Noritake, M. Banbara, T. Soh, N. Tamura, and K. Inoue. Constraint modeling and SAT encoding of the packing array problem.Computer Software, 31:1_116–1_130, 2014

    2014

  28. [36]

    Novikov, N

    A. Novikov, N. Vu, M. Eisenberger, E. Dupont, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery, 2025. DeepMind

    2025

  29. [37]

    OpenAI o3-mini system card, 2025

    OpenAI. OpenAI o3-mini system card, 2025. https://cdn.openai.com/o3-mini-system-card- feb10.pdf

    2025

  30. [38]

    Petrov, J

    I. Petrov, J. Dekoninck, L. Baltadzhiev, M. Drencheva, K. Minchev, M. Balunovi´c, N. Jovanovi´c, and M. Vechev. Proof or bluff? Evaluating LLMs on 2025 USA Math Olympiad, 2025. arXiv:2503.21934

    2025 arXiv

  31. [39]

    Romera-Paredes, M

    B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, et al. Mathematical discoveries from program search with large language models.Nature, 625(7995):468–475, 2024

    2024

  32. [40]

    C. D. Rosin. Using code generation to solve open instances of combinatorial design problems,

  33. [41]

    Shehadeh, W

    M. Shehadeh, W. Kingsford, and F. R. Kschischang. New difference triangle sets by an FPGA-based search technique, 2025. arXiv:2502.19517

    2025 arXiv

  34. [42]

    Shokri and L

    K. Shokri and L. Moura. New families of strength-3 covering arrays using linear feedback shift register sequences.Journal of Combinatorial Designs, 2025. 10

    2025

  35. [43]

    J. Stardom. Metaheuristics and the search for covering and packing arrays. Master’s thesis, Simon Fraser University, 2001

    2001

  36. [44]

    Stroebl, S

    B. Stroebl, S. Kapoor, and A. Narayanan. Inference Scaling fLaws: The limits of LLM resampling with imperfect verifiers, 2024. arXiv:2411.17501

    2024

  37. [45]

    W. Sun, S. Feng, S. Li, and Y . Yang. CO-Bench: Benchmarking language model agents in algorithm search for combinatorial optimization, 2025. arXiv:2504.04310

    2025 arXiv

  38. [46]

    Swirszcz, A

    G. Swirszcz, A. Z. Wagner, G. Williamson, S. Blackwell, B. Georgiev, A. Davies, A. Eslami, S. Racaniere, T. Weber, and P. Kohli. Advancing geometry with AI: Multi-agent generation of polytopes, 2025. arXiv:2502.05199

    2025 arXiv

  39. [47]

    Torres-Jimenez and E

    J. Torres-Jimenez and E. Rodriguez-Tello. New bounds for binary covering arrays using simulated annealing.Information Sciences, 185(1):137–152, 2012

    2012

  40. [48]

    Torres-Jimenez, H

    J. Torres-Jimenez, H. Avila-George, and I. Izquierdo-Marquez. A two-stage algorithm for combinatorial testing.Optimization Letters, 11(3):457–469, 2017

    2017

  41. [49]

    Torres-Jimenez, I

    J. Torres-Jimenez, I. Izquierdo-Marquez, and H. Avila-George. Methods to construct uniform covering arrays.IEEE Access, 7:42774–42797, 2019

    2019

  42. [50]

    A. Z. Wagner. Constructions in combinatorics via neural networks, 2021. arXiv:2104.14516

    2021 arXiv

  43. [51]

    Weindel and R

    F. Weindel and R. Heckel. LLM-guided search for deletion-correcting codes, 2025. arXiv:2504.00613

    2025 arXiv

  44. [52]

    White, S

    C. White, S. Dooley, M. Roberts, et al. LiveBench: A challenging, contamination-limited LLM benchmark. InICLR, 2025

    2025

  45. [53]

    K. Yang, A. Swope, A. Gu, R. Chalamala, P. Song, S. Yu, S. Godil, R. J. Prenger, and A. Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. NeurIPS, 2024

    2024

  46. [54]

    Balanced Incomplete Block Design

    H. Ye, J. Wang, Z. Cao, F. Berto, C. Hua, H. Kim, J. Park, and G. Song. ReEvo: Large language models as hyper-heuristics with reflective evolution. InNeurIPS, 2024. 11 A Appendix: Definitions Used for Prompting 12 A"Balanced Incomplete Block Design"BIBD(v,b,r,k,L) is a pair (V...

    2024

  47. [55]

    Each element appears R=p1 + 2*p2 times altogether, with multiplicity one in exactly p1 blocks and multiplicity two in exactly p2 blocks

  48. [56]

    Costas Array

    Every pair of distinct elements appears L times; that is, if m_{vb} is the multiplicity of the v’th element in the b’th block, then for every pair of distinct elements v and w, sum_{b=1}^{B} m_{vb} m_{wb} = L. The BTD is represented by a V by B incidence matrix with elements i...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.