Pith. sign in

REVIEW 2 major objections 5 minor 4 cited by

Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An open-source 8B LLM, used as both generator and scorer, solves 71.6% of ARC-AGI tasks.

desk verdict Strong open-source ARC-AGI result, but the fractional solved-task count needs clarification before the headline is fully interpretable. read the letter →

arxiv 2505.07859 v2 pith:VJIBQPNW submitted 2025-05-08 cs.CL cs.AIcs.LG

Daniel Franzen , Jan Disselhoff , David Hartmann This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords ARC-AGIabstractionandreasoningproductofexpertstest-timetrainingdepth-firstsearchdataaugmentationLLMgeometric-meanensemble
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a single open-source 8-billion-parameter language model, fine-tuned on synthetically generated ARC-AGI training puzzles, can reach state-of-the-art performance on the benchmark by being used twice: first as a generator of candidate answers through threshold-pruned depth-first search, and then as a scorer that re-ranks candidates by the product of their probabilities across many augmented views of the problem. The method reports 71.6% accuracy on the public ARC-AGI evaluation set, above the 60.2% average human score and above prior open-source approaches. This matters because it suggests that abstract reasoning on ARC does not require massive proprietary models: the same dual-role trick, test-time training, and cheap hardware yield competitive results at roughly two cents of inference per task.

What carries the argument

The load-bearing object is the geometric-mean ensemble of LLM likelihoods over augmentation-preserved views of a puzzle, called a product of experts. For each candidate solution $s$, the model computes $\hat{P}(\phi_j(s)|\phi_j(p))$ under every transformation $\phi_j$ and multiplies them, forcing a candidate to be plausible from every perspective at once and suppressing spurious high-confidence errors. The companion mechanism is DFS-based candidate generation: a depth-first search over solution tokens prunes any partial path whose accumulated log-probability falls below a threshold $T$, producing a small, diverse set $C_{p,T}$ of high-probability candidates cheaply. A 64-token vocabulary (one token per grid-cell color plus delimiters) and test-time fine-tuning on each task's examples supply the conditions under which the likelihood signal is trustworthy.

What would settle it

Run the pipeline on a held-out set of puzzles where the correct answer is known, sample $C_{p,T}$ with $T=9\%$ so the correct solution is almost always present, and check whether the product-of-experts score picks the correct candidate. If a substantial share of tasks have a wrong candidate with higher aggregate score than the correct one, the ranking assumption fails; an even sharper test is to use augmentations that are not true symmetries (e.g., color remappings that break a task's semantics) and show that PoE then selects incorrect candidates.

Watch

Extended reading notes

Core claim

The central discovery is that next-token likelihood under a fine-tuned LLM is a usable ranking signal for ARC solutions, provided it is aggregated across transformations. The paper defines a candidate set $C_{p,T}$ of solutions pruned by a probability threshold $T$ via depth-first search, then scores each candidate with $\text{score}_{\text{agg}}(s)=\prod_{\phi_j\in\Phi}\hat{P}(\phi_j(s)|\phi_j(p))$, a product-of-experts (geometric-mean) ensemble over D8 symmetries, color permutations, and example orderings. It proves that the KL divergence of this ensemble from the true solution distribution is bounded by the average single-augmentation KL divergence plus a non-positive log-normalization term, so the ensemble is in expectation at least as good as a randomly chosen augmentation. On the public ARC-AGI evaluation set this pipeline solves 286.5 of 400 tasks (71.6%), generalizes to 73.3% on ConceptARC, and reaches 53% on a Sudoku benchmark; in the reported experiments, when the correct Sudoku solution is sampled, the ensemble selects it 100% of the time.

Load-bearing premise

The load-bearing premise is that the LLM's next-token probabilities rank solution quality reliably, so that DFS pruning at threshold $T$ and the product-of-experts score select the correct output; the paper does not separately calibrate or test this ranking assumption apart from the final benchmark score.

Editorial extensions

If this is right

  • An open-source 8B model with LoRA fine-tuning, test-time training, and the generate-then-rescore pipeline matches or beats the average human on ARC-AGI public tasks.
  • The same method transfers to other ARC-like benchmarks (ConceptARC) and structured domains (Sudoku), suggesting a recipe rather than a benchmark-specific patch.
  • Inference cost drops to about $0.02 per task on a single Nvidia RTX 4090, versus the reported $17 per task for the closed-source o3 system.
  • Selecting by product of experts outperforms selecting by mean probability, max probability, or single-augmentation probability in all reported sampling regimes.
  • Threshold $T$ controls the coverage/selectivity tradeoff: lower $T$ finds more correct solutions but floods the candidate set with false positives; moderate $T$ around 9% gives the best two-guess accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The augmentation-product trick should transfer to any reasoning domain with cheap semantic-preserving transformations, such as shuffled program syntax, paraphrased math word problems, or rotated visual grids; disagreement across views is what makes the product filter false positives.
  • The KL bound suggests that the ensemble's benefit grows exactly when the LLM is inconsistent across augmentations, so deliberately diverse problem representations could be engineered rather than taken as a given.
  • Because the whole pipeline leans on next-token probability as a quality proxy, calibrating those probabilities (e.g., with temperature or labeled validation tasks) could further close the gap to the upper-bound coverage line in Figure 4.
  • A natural extension the paper leaves implicit is applying the DFS+PoE scheme to induction-style solutions (inferred programs) instead of only transduction-style grid outputs, potentially combining both routes on the same puzzle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a pipeline for solving ARC-AGI tasks with an 8B open-source LLM. The model is initially fine-tuned on RE-ARC (generated from the official training tasks), then test-time fine-tuning is performed on each task's examples. Candidate solutions are generated by a depth-first search with a probability threshold T, and the best candidates are selected by a product-of-experts score computed across 16 augmented views of the problem. The main claim is 71.6% (286.5/400) two-guess accuracy on the public ARC-AGI evaluation set, which would be state of the art among open-source approaches and above the average human score. Additional results are reported on ConceptARC and a Sudoku benchmark.

Significance. If the headline result is reproducible, it is a substantial advance for open-source LLMs on ARC-AGI: an 8B model with roughly two cents of inference per task outperforms cited open-source baselines and the average human. The paper's strengths include a systematic ablation (Table 2), a comparison of sampling and selection strategies (Table 3), external validation on ConceptARC and Sudoku, and a simple theoretical justification for product-of-experts ranking. The authors state that code and the final model are publicly available. However, the central number's internal inconsistency and the evaluation-set tuning of the DFS threshold currently prevent the paper from being taken at face value.

major comments (2)
  1. [Abstract and Section 2] The reported result "71.6% (286.5/400 solved tasks)" in the Abstract is incompatible with the binary per-task definition of "solved" given in Section 2, where a task is considered solved when the model produces the accurate output within a maximum of two attempts. A count over 400 tasks must be an integer under that definition, unless partial credit or averaging over randomized runs is used, but the paper nowhere defines such a scoring rule. This matters because Table 1 compares the 71.6% against baselines and the human average that are computed on a binary per-task basis. Please clarify the exact scoring protocol and, if a partial-credit or averaged metric is used, recompute the comparisons on the binary metric; alternatively, correct the count.
  2. [Section 5.3] The DFS probability threshold T is selected by inspecting the public evaluation set ("We found that values between T = 5% to T = 20% provided a reasonable compromise... Our final results are calculated using T = 9%"). Since the reported 71.6% is measured on that same set, the headline result is affected by evaluation-set tuning. This is a methodological concern for the claimed state-of-the-art comparison and for the statement that the method generalizes to ConceptARC, where the same hyperparameters are used. Please disclose this explicitly and, if possible, report the model's performance with T chosen on a development set (e.g., a held-out subset of the ARC training tasks) or show that the result is robust across the reported T values.
minor comments (5)
  1. [Section 5.3] The sentence "This function is monotonically increasing in T" is reversed relative to the data in Table 3: as T increases from 0.5% to 20%, the fraction of candidate sets containing the correct solution decreases from 83.5% to 73.5%. Please correct the direction or rephrase to describe the -log(prob) threshold used in Figure 4.
  2. [Appendix C] In Step 2 of the proof of Theorem C.1, the line reading "Z = ∑_s ∑_j (1/m) \hat P_j(s) = 1" should be written as an inequality (Z ≤ ... ≤ 1); as typeset, the equality is a mathematical error that could confuse readers about the argument.
  3. [General] The paper states that the final model and code are publicly available on GitHub but does not provide a URL or repository identifier anywhere in the text; please add the link so the reproducibility claim is actionable.
  4. [Abstract and Introduction] The abstract uses "2ct per task" and the introduction uses "0.02$ per task"; please use a consistent currency notation and include the explicit conversion to cents.
  5. [Table 1] The "Open Source" column has a question mark for "Avg. Human", which is not a meaningful open/proprietary classification; consider marking it as N/A or removing the column for that row.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark score is externally evaluated, the PoE theorem is a self-contained mathematical inequality, and no load-bearing self-citations are present.

full rationale

The paper's central claim is an empirical benchmark score on the public ARC-AGI evaluation set, not a quantity derived from its own assumptions. The training data (RE-ARC) is procedurally generated from the official ARC training tasks, not from the evaluation tasks; the only evaluation-set information used at inference is each task's own input-output examples during test-time training, which is standard transductive inference and does not reveal the test output. The product-of-experts scoring is justified by Theorem 4.1, a self-contained KL-divergence inequality with a proof in Appendix C; it is a mathematical identity about geometric-mean pooling of probability distributions, not a fitted or self-referential claim. The DFS threshold T is described as a hyperparameter chosen for a runtime/accuracy trade-off; although it was selected with the public evaluation set in view, this is a benchmark-tuning caveat rather than a case where a prediction reduces to its inputs. There are no load-bearing self-citations: the paper cites Hinton for the PoE concept, Akyurek et al. for TTT, and Hodel for RE-ARC, none of which are the present authors' prior work, and none is invoked to forbid alternative explanations. The internal inconsistency of reporting '286.5/400 solved tasks' against the paper's own binary two-attempt definition is a metrics/reporting issue, not a circularity. Overall, the derivation chain is self-contained: candidates are generated from the model's probabilities, rescored by the same probabilities under augmentations, and judged against external ground truth, with no equation defined in terms of the quantity it purports to predict.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The pipeline assumes ARC's symmetries and the reliability of LLM token probabilities, but introduces no new physical or conceptual entities. The only numerically tuned quantity central to the claim is the DFS threshold T, chosen on the public evaluation set.

free parameters (2)
  • DFS probability threshold T = 9% (alternative T=0.5% yields 71.8%; T=5-20% claimed reasonable)
    T controls candidate set size and runtime; the final value was selected after inspecting public eval results (Section 5.3), so the reported 71.6% is partly in-sample.
  • Number of augmentations per task = 16 for generation, 16 for scoring
    Hand-chosen without a reported sensitivity analysis; the count affects both candidate diversity and inference cost.
assumptions (5)
  • domain assumption For every valid augmentation phi in Phi, the solution distribution is invariant: P(s|p) = P(phi(s)|phi(p)).
    Stated in Section 3 as defining Phi. The PoE scoring and training-augmentation strategy assume ARC tasks satisfy D8 symmetry, color permutation, and example-order invariance. If some tasks violate this, candidates could be mis-ranked.
  • domain assumption The LLM's next-token probabilities, after fine-tuning and test-time training, approximate the true solution distribution well enough that high-probability candidates are more likely correct.
    Used to define C_{p,T} and score_agg in Section 4. The pipeline provides no calibration or independent check of this ranking assumption beyond the reported benchmarks.
  • domain assumption RE-ARC generated tasks faithfully represent the official 400 training tasks and do not leak evaluation tasks into training.
    Section 5.2 trains only on RE-ARC to avoid conceptual leakage; the faithfulness of Hodel's generators is taken for granted.
  • domain assumption Test-time training on a single task's examples improves the model's solution distribution for that task without harmful overfitting on the few available examples.
    The TTT step in Section 5.2 is central to the pipeline; the paper reports that it more than doubles baseline accuracy but does not analyze failure cases.
  • standard math The geometric-mean inequality log Z <= 0 and the KL divergence identity used in Theorem 4.1 are valid.
    Appendix C relies on standard properties of arithmetic/geometric means and KL divergence; the result is mathematically correct but not machine-checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective." pith.science (2026). https://pith.science/paper/VJIBQPNW

@misc{pith2026250507859,
  author       = {Pith},
  title        = {Pith review of: Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJIBQPNW}},
  note         = {Machine review of arXiv:2505.07859}
}
read the original abstract

The Abstraction and Reasoning Corpus (ARC-AGI) poses a significant challenge for large language models (LLMs), exposing limitations in their abstract reasoning abilities. In this work, we leverage task-specific data augmentations throughout the training, generation, and scoring phases, and employ a depth-first search algorithm to generate diverse, high-probability candidate solutions. Furthermore, we utilize the LLM not only as a generator but also as a scorer, using its output probabilities to select the most promising solutions. Our method achieves a score of 71.6% (286.5/400 solved tasks) on the public ARC-AGI evaluation set, demonstrating state-of-the-art performance among publicly available approaches. While concurrent closed-source work has reported higher scores, our method distinguishes itself through its transparency, reproducibility, and remarkably low inference cost, averaging only around 2ct per task on readily available hardware (we assume a price of 36ct/hour for a Nvidia 4090 GPU).

Figures

Figures reproduced from arXiv: 2505.07859 by the authors.

Figure 1
Figure 1. Example of a typical ARC-AGI task. illustrated in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our standard tokenization approach. Note that we use one token per cell instead of compressing the problem more. We also try to not include any unnecessary delimiters. The Pre-prompt (the alphabet in upper then lower case, i.e. “A...Za...z”) is only in￾cluded for the first example. Depending on the model and run there might be some small changes to the pre-prompt and input/output prefix tokens. The Original ARC Data… view at source ↗
Figure 3
Figure 3. Number of solutions found by various sampling algo￾rithms as a function of runtime. The different values for each sampling variant are calculated using 1 (identity), 2 (reflections), 4 (rotation), 8 (reflections+rotation) and 16 augmentations. Addi￾tionally, colors and the order or examples are randomly permuted in each augmented version of a task. For almost any runtime budget, we find that a DFS variant discovers … view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Comparing the rank of the correct solution using the gen￾erative model Pˆ and the ensemble selection P among candidates Cp,T . If possible, our ensemble almost always improves the rank of the correct solution, increasing the chance of selecting it. For readability we c…
Figure 6
Figure 6. Figure 6: Results of the Sudoku experiments (plot equivalent to [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think Through a Bottleneck: Hourglass Reasoning for Rigorous Induction

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Strict stage isolation that passes only a compressed symbolic schema and rule between LLM calls improves few-shot inductive reasoning more than self-refinement or explicit verbalization alone.

  2. Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Explorer-Definer and Reflective Orchestrator harnesses raise DeepSeek V3.2 from 15.5% to 67.25% pass@2 on ARC-AGI-1 public eval at $0.25–$0.62 per task without ARC-specific training.

  3. Context Tuning for In-Context Optimization

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Context Tuning initializes a trainable prompt or KV-cache prefix from the few-shot demonstrations and refines it by gradient descent, matching Test-Time Training accuracy on MMLU, BBH, and ARC at up to 2-3x less train...

  4. Channel-Wise MLPs Improve the Generalization of Recurrent Convolutional Networks

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Adding a gated channel-wise MLP to a recurrent convolutional network raises median exact-match accuracy on 185 Re-ARC tasks from 78.75% to 92.19% in-distribution and from 2.34% to 14.58% on harder out-of-distribution tasks.

Reference graph

Works this paper leans on

30 extracted references · 15 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    The surprising effectiveness of test-time training for abstract reasoning, 2024

    Akyürek, E., Damani, M., Qiu, L., Guo, H., Kim, Y., and Andreas, J. The surprising effectiveness of test-time training for abstract reasoning, 2024. URL https://arxiv.org/abs/2411.07279

    arXiv 2024

  3. [3]

    and Li, Y

    Allen - Zhu, Z. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL https://openreview.net/forum?id=5x788rqbcj

    work page 2024

  4. [4]

    and Li, Y

    Allen - Zhu, Z. and Li, Y. Physics of language models: Part 3.2, knowledge manipulation, 2025. URL https://openreview.net/forum?id=oDbiL9CLoS

    work page 2025

  5. [5]

    O pen A I o3 B reakthrough H igh S core on A R C - A G I - P ub --- arcprize.org

    arcprize.org. O pen A I o3 B reakthrough H igh S core on A R C - A G I - P ub --- arcprize.org. https://arcprize.org/blog/oai-o3-pub-breakthrough, 2025. [Accessed 25-01-2025]

    work page 2025

  6. [6]

    H ow I came in first on A R C - A G I - P ub using S onnet 3.5 with E volutionary T est-time C ompute --- jeremyberman.substack.com

    Berman, J. H ow I came in first on A R C - A G I - P ub using S onnet 3.5 with E volutionary T est-time C ompute --- jeremyberman.substack.com. https://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi, 2024. [Accessed 25-01-2025]

    work page 2024

  7. [7]

    and Durrett, G

    Bostrom, K. and Durrett, G. Byte pair encoding is suboptimal for language model pretraining, 2020. URL https://doi.org/10.18653/v1/2020.findings-emnlp.414

    doi:10.18653/v1/2020.findings-emnlp.414 2020

  8. [8]

    On the measure of intelligence

    Chollet, F. On the measure of intelligence. CoRR, abs/1911.01547, 2019. URL http://arxiv.org/abs/1911.01547

    arXiv 1911

Show all 30 references
  1. [9]

    C ommunity I nterview J ack C ole -- L ab42 --- lab42.global

    Cole, J. C ommunity I nterview J ack C ole -- L ab42 --- lab42.global. https://lab42.global/community-interview-jack-cole/, 2024. [Accessed 29-01-2025]

    2024

  2. [10]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al - Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roz...

    2024 doi arXiv

  3. [11]

    G etting 50\ --- redwoodresearch.substack.com

    Greenblatt, R. G etting 50\ --- redwoodresearch.substack.com. https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt, 2024. [Accessed 25-01-2025]

    2024

  4. [12]

    Hinton, G. E. Products of experts. In 9th International Conference on Artificial Neural Networks: ICANN '99 . IEE, 1999

    1999

  5. [13]

    Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Comput., 14 0 (8): 0 1771--1800, 2002. doi:10.1162/089976602760128018. URL https://doi.org/10.1162/089976602760128018

    2002 doi

  6. [14]

    Addressing the abstraction and reasoning corpus via procedural example generation, 2024

    Hodel, M. Addressing the abstraction and reasoning corpus via procedural example generation, 2024. URL https://arxiv.org/abs/2404.07353

    2024 arXiv

  7. [15]

    J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen - Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  8. [16]

    Testing o1-preview on ARC-AGI , 2024

    Kamradt, G. Testing o1-preview on ARC-AGI , 2024. URL https://www.kaggle.com/code/gregkamradt/testing-o1-preview-on-arc-agi. Accessed: 2024-11-10

    2024

  9. [17]

    K., Lake, B

    LeGris, S., Vong, W. K., Lake, B. M., and Gureckis, T. M. H-arc: A robust estimate of human performance on the abstraction and reasoning corpus benchmark, 2024. URL https://arxiv.org/abs/2409.01374

    2024 arXiv

  10. [18]

    Li, W., Xu, Y., Sanner, S., and Khalil, E. B. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects, 2024. URL https://arxiv.org/abs/2410.06405

    2024 arXiv

  11. [19]

    M., Tang, H., Zheng, W., Pu, Y., and Ellis, K

    Li, W., Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Zheng, W., Pu, Y., and Ellis, K. Combining induction and transduction for abstract reasoning, 2025. URL https://openreview.net/forum?id=UmdotAAVDe

    2025

  12. [20]

    F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P

    Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. Trans. Assoc. Comput. Linguistics, 12: 0 157--173, 2024. doi:10.1162/TACL\_A\_00638. URL https://doi.org/10.1162/tacl\_a\_00638

    2024 doi

  13. [21]

    V., and Mitchell, M

    Moskvichev, A., Odouard, V. V., and Mitchell, M. The conceptarc benchmark: Evaluating understanding and generalization in the ARC domain. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=8ykyGbtt2q

    2023

  14. [22]

    79fb03f4 test is unsolvable, water flow · I ssue \#100 · fchollet/ A R C - A G I --- github.com

    Neoneye. 79fb03f4 test is unsolvable, water flow · I ssue \#100 · fchollet/ A R C - A G I --- github.com. https://github.com/fchollet/ARC-AGI/issues/100, 2024. [Accessed 30-01-2025]

    2024

  15. [23]

    Radcliffe, D. G. 3 million sudoku puzzles with ratings, 2020. URL https://www.kaggle.com/dsv/1495975

    2020

  16. [24]

    Puzzle id: 0d87d2a6 result is ambiguous

    RubenKelevra. Puzzle id: 0d87d2a6 result is ambiguous. https://github.com/fchollet/ARC-AGI/issues/149, 2024. [Accessed 30-01-2025]

    2024

  17. [25]

    Sudoku-bench: Evaluating creative reasoning with sudoku variants, 2025

    Seely, J., Imajuku, Y., Zhao, T., Cetin, E., and Jones, L. Sudoku-bench: Evaluating creative reasoning with sudoku variants, 2025. URL https://arxiv.org/abs/2505.16135

    2025 arXiv

  18. [26]

    Singh, A. K. and Strouse, D. Tokenization counts: the impact of tokenization on arithmetic in frontier llms. CoRR, abs/2402.14903, 2024. doi:10.48550/ARXIV.2402.14903. URL https://doi.org/10.48550/arXiv.2402.14903

    2024 doi arXiv

  19. [27]

    T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P

    Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Llm pruning and distillation in practice: The minitron approach, 2024. URL https://arxiv.org/abs/2408.11796

    2024 arXiv

  20. [28]

    Y., and Huang, Z

    Sun, K., Qi, P., Zhang, Y., Liu, L., Wang, W. Y., and Huang, Z. Tokenization consistency matters for generative models on extractive NLP tasks. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Dece...

    2023 doi

  21. [29]

    A., and Hardt, M

    Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volum...

    2020

  22. [30]

    Wind, J. S. A bstraction and R easoning C hallenge --- kaggle.com. https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/discussion/154597, 2020. [Accessed 30-01-2025]

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.