Pith. sign in

REVIEW 3 major objections 6 minor 79 references

From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Progressively prompting LLMs with ontology-structured core knowledge priors improves their abstract reasoning and generalization on the ARC benchmark, by about five absolute percentage points and up to 64.52% relative across four…

desk verdict A useful systematic evaluation of reasoning LLMs on ARC, but the headline KAAR gain is confounded by unmatched solver compute; the fix is straightforward. read the letter →

arxiv 2505.17482 v1 pith:ZA4TQASZ submitted 2025-05-23 cs.AI cs.CL

Chao Lei , Nir Lipovetzky , Krista A. Ehinger , Yanchuan Chang This is my paper
classification cs.AIcs.CL
keywords AbstractionandReasoningCorpuslargelanguagemodelsknowledgeaugmentationcorepriorsprogramsynthesisgeneralizationobject-centricrepresentationchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that reasoning-oriented LLMs can be made better at abstract reasoning and generalization by prompting them with structured core knowledge priors in stages, rather than all at once. The evaluation target is the Abstraction and Reasoning Corpus (ARC), where each task demands inferring a transformation from a handful of input-output pixel grids and applying it to new inputs. The authors build KAAR, which organizes priors about objects, geometry, numbers, and goals into a three-level ontology and re-runs a strong code-generation baseline (RSPC) after each level. Across four LLMs, KAAR improves test accuracy by about five absolute percentage points over RSPC, with the relative gain reaching 64.52% for the weakest model. A sympathetic reader would take this as evidence that staged, human-like prior knowledge can expand what LLMs generalize to, without fine-tuning or bigger models.

What carries the argument

The central object is the KAAR ontology of core knowledge priors, grouped into three dependency levels: objectness (image abstractions such as whole image, middle splits, multi-lines, and 4/8-connected components), geometry and topology plus numbers and counting (component attributes like size, shape, symmetry, holes, alongside relational and frequency statistics), and goal-directedness (ten predefined actions such as color change, movement, and extension, each with a schema for implementation details). After augmenting each level, KAAR invokes the RSPC backbone to generate a plan-guided Python program and checks it against training instances. The selection mechanism is the 'first abstraction whose solution solves all training instances'; if no abstraction succeeds, the code that passes the most training instances is chosen and evaluated on the test set.

What would settle it

Run a matched-compute control: give RSPC the same total solver budget KAAR spends (for instance, three invocations of up to four iterations per abstraction, or the equivalent number of repeated samples) but with no knowledge prompts, and compare test accuracy on the same 400 ARC tasks; if the gap shrinks toward zero, the staged priors are not the cause. A second falsifier is to inject staged but content-free priors of equal length and see whether the gain persists.

Watch

Extended reading notes

Core claim

On the ARC public evaluation set of 400 problems, KAAR—which encodes core knowledge priors into a dependency-based ontology and augments the model after each of three levels while invoking the repeated-sampling planning-aided code solver (RSPC) for up to four iterations per level—reports test accuracies of 35.0% with GPT-o3-mini, 21.75% with Gemini-2.0, 21.0% with QwQ-32B, and 12.75% with DeepSeek-R1-70B, compared with 30.75%, 16.75%, 14.25%, and 7.75% for RSPC alone. The same pattern holds for training-instance accuracy and for combined train-and-test accuracy, leading the authors to describe the improvement as model-agnostic. The paper also reports that repeated-sampling solvers beat refinement-based solvers on ARC, contrary to common findings on conventional code-generation benchmarks, and that planning-aided code generation closes much of the generalization gap between training and test accuracy.

Load-bearing premise

The central claim rests on the assumption that the extra solver invocations KAAR makes across its abstraction levels do not by themselves account for the measured gains.

Editorial extensions

If this is right

  • Across all four evaluated LLMs, KAAR beats RSPC on test, training, and combined accuracy, with absolute test gains of 4.25 to 6.75 points and relative gains up to 64.52%.
  • The ablation study shows that injecting all priors at once (KAAR*) costs 2.5 to 3.75 points compared with stage-wise injection, supporting the paper's claim that progressive augmentation reduces interference.
  • Accuracy gains concentrate in movement tasks and in images with average size below 400 pixels; extension tasks and images larger than 20 by 20 remain largely unsolved.
  • Repeated sampling with planning-aided code generation generalizes better on ARC than refinement-based approaches, so the best solver choice for ARC-type tasks differs from typical code-generation benchmarks.
  • KAAR also raises cross-model coverage: the proportion of problems solved by the weakest model that are also solved by the strongest rises from 0.22 to 0.34, suggesting shared reasoning improvements across models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because KAAR invokes the solver up to three times per abstraction (four iterations each) while the baseline gets twelve iterations total, a matched-compute control would be needed to separate the effect of the knowledge prompts from the effect of extra solver calls; this is my inference, not a claim the paper makes.
  • The first-abstraction-that-passes-training heuristic is a likely ceiling on generalization: the paper itself shows cases where it stops at the no-abstraction stage and reproduces the baseline's overfit solution, which suggests learning to pick abstractions from the task itself is the next bottleneck.
  • The same ontology machinery could be ported to other hierarchy-dependent reasoning domains such as robotic task planning or visual question answering, where object-level priors and goal-directed action schemas play a comparable role.
  • A natural stress test for the paper's hypothesis is to replace the knowledge priors with matched-length random or irrelevant structured text: if the staged prompts are what matter, the random control should not reproduce the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper evaluates recent reasoning-oriented LLMs (GPT-o3-mini, Gemini-2.0-Flash-Thinking, DeepSeek-R1-70B, QwQ-32B) on the ARC public evaluation set of 400 tasks, comparing nine solver configurations formed from direct generation, repeated sampling, and refinement, with plan, standalone-code, or planning-aided-code outputs. It identifies repeated-sampling planning-aided code generation (RSPC) as the strongest backbone and introduces KAAR, which augments the LLM prompt with core knowledge priors organized into a three-level ontology derived from the authors' GPAR system, invoking RSPC after each augmentation stage. The paper reports that KAAR outperforms RSPC by roughly 5% absolute across all evaluated LLMs, with the largest relative gain of 64.52% on DeepSeek-R1-70B, and interprets this as evidence that progressively prompting with ontology-structured priors improves abstract reasoning and generalization.

Significance. If the reported gains were cleanly attributable to the knowledge priors, this would be a useful empirical contribution to LLM-based ARC solving: the paper provides a systematic comparison over the full public evaluation set, an explicitly described staged-prompting protocol, detailed failure analysis, and complete prompts and pseudocode in the appendix, and it honestly documents overfitting cases in Appendix A.9. The main result, however, is currently confounded by an unequal compute budget between KAAR and RSPC, so the causal claim in the abstract and Section 5 is not yet supported by the evidence presented. The systematic solver comparison and the public evaluation data are valuable independent of this issue, and the staged augmentation idea is worth exploring further once the compute-matching problem is addressed.

major comments (3)
  1. [Section 5; Algorithm 1 (Appendix A.5)] The statement that KAAR allows the solver backbone (RSPC) up to 4 iterations per invocation, totaling 12 iterations, consistent with the non-augmented setting, is not a correct description of KAAR's total budget. Algorithm 1 invokes SolverBackbone three times per abstraction (Lines 6, 15, and 19), each with t=4, and the outer loop (Line 3) continues over abstractions until a solution passes Ir. KAAR's total solver-iteration budget is therefore 12×K, where K is the number of abstractions tried, plus the token cost of augmentation. RSPC, by contrast, receives exactly 12 iterations in total (Section 3). Because no matched-compute RSPC baseline is reported (Table 4 reports only KAAR token usage, and Figure 8 shows RSPC accuracy still increasing with iteration count until roughly iteration 8), the ~5% absolute gains in Table 2 may be partly attributable to additional test-time search rather than to knowledge augmentation. The central causal claim of the abstract and Section 5 requires a matched-compute control, for example RSPC run for 12×K iterations or with an equal token budget, before the gain can be attributed to the augmentation itself.
  2. [Section 5; Appendix A.9] The abstraction-selection rule reported in Section 5, which picks the first abstraction whose solution solves all training instances Ir (and otherwise selects the code passing the most training instances), is a training-set selection heuristic that can overfit. The paper itself documents two cases (695367ec and b1fc8b8e) in Appendix A.9 where KAAR starts with the 'no abstraction' variant, reproduces the same overfit RSPC solution that passes Ir, and terminates without exploring other abstractions. Consequently, the aggregate It figures in Table 2 average over cases in which KAAR is identical to RSPC and cases in which it consumes additional compute and abstractions. The paper should report the distribution of the abstraction index K at which each task terminates, and the proportion of tasks resolved by the 'no abstraction' variant, so that the contribution of the augmentation protocol can be separated from the baseline solver's behavior.
  3. [Table 2; Appendix A.4] Accuracy differences are reported without any measure of uncertainty or repeated runs. The open-source models are sampled with temperature 0.6, top-p 0.95, and top-k 40 (Appendix A.4), so the results are stochastic; a 5% absolute difference over 400 tasks may be within sampling noise. The paper should provide multiple seeds or confidence intervals (for example, a bootstrap over the 400 tasks) for the RSPC versus KAAR comparison before claiming that KAAR consistently outperforms RSPC across all LLMs.
minor comments (6)
  1. [Table 1] Some entries in the table are run together without separators (for example, '35.5052.5035.50' in the GPT-o3-mini Ir row), which makes the table very hard to read; adding spacing or visible column separators would improve clarity.
  2. [Figure 8] The caption describes the plot as 'Variance in accuracy with increasing iterations,' but the figure actually plots accuracy as a function of iteration count; the label 'variance' is a misnomer and should be replaced with something like 'Accuracy versus iterations.'
  3. [Appendix A.13, Prompt 2] The prompt contains a duplicated article: 'the the output images should be outputted as' should be 'the output images should be outputted as.'
  4. [Table 2] The column header 'Acc∆γAcc∆γAcc∆γ' lacks visible separation between the three metric groups, making it difficult to map columns to Ir, It, and Ir&It; use clearer headers with distinct column boundaries.
  5. [Algorithm 1, Line 18] The line 'AugmentGoalPriors←(M,Q,abs)' appears to be missing a function-call syntax; it should probably read 'AugmentGoalPriors(M,Q,abs)' or use an explicit assignment to a result variable.
  6. [Section 4 and Table 5] Listing 'no abstraction' among the 'abstractions' is confusing because it is explicitly the absence of abstraction; the paper should consistently distinguish this baseline variant from the actual abstraction methods, for instance by calling it 'no-abstraction baseline' throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: KAAR's gains are measured on held-out ARC test instances, and the reliance on the authors' prior GPAR work supplies method components, not the reported result.

full rationale

The paper's central claim is an empirical comparison: KAAR is reported to outperform RSPC on the ARC public evaluation set. The derivation chain is not circular because the test accuracy is obtained by executing LLM-generated code on held-out test inputs; the augmented priors are computed from the task's training inputs and test inputs via image-processing algorithms, not from the test outputs being predicted. The abstraction-selection rule ('first abstraction whose solution solves all training instances') is a training-set model-selection heuristic, and the paper separately reports test accuracy, so the prediction is not fitted to the target by construction. KAAR does inherit its priors from GPAR [10], a paper sharing authors with the present work, but that self-citation supplies the method's components and vocabulary rather than the empirical outcome: no uniqueness theorem is imported, no fitted parameter is renamed as a prediction, and the reported It numbers are not equal to any training-set fit. The skeptic's compute-budget asymmetry (KAAR can run up to 12 solver iterations per abstraction across multiple abstractions, while RSPC is limited to 12 total) is a potential threat to the causal attribution of the gain, but it is an experimental-validity concern, not circularity. The paper also self-documents failure cases and the 'no abstraction' degradation (Appendix A.9), which further shows the comparison is not definitionally forced. Therefore no circular step is exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 3 invented entities

The central claim rests on several hand-chosen design elements: the abstraction set and its order, the per-stage iteration budget, the three-level ontology, and the selection heuristic based on training-set success. These are not fitted numerical parameters in the usual sense, but they are free choices that could be tuned to the evaluation set. The key axioms are the ARC domain assumption that core knowledge priors suffice, and the assumption that the image-processing abstractions faithfully represent objectness.

free parameters (3)
  • solver iterations per augmentation stage = 4
    Algorithm 1 sets t=4, giving 12 iterations per abstraction. Chosen by hand and not justified against other budgets.
  • abstraction ordering and count = 19 abstractions ordered by simplicity
    Table 5 defines the set and order; the selection of the first abstraction that solves Ir is a heuristic that affects results.
  • sampling temperature, top-p, top-k = 0.6, 0.95, 40 for open-source models
    Appendix A.4; not reported for proprietary models, and no seed or replication information is given.
assumptions (4)
  • domain assumption ARC tasks are solvable using core knowledge priors (objectness, geometry/topology, numbers/counting, goal-directedness).
    Adopted from Chollet [5]; the paper relies on this to justify the prior set used in KAAR.
  • domain assumption Connected-component and other image abstractions capture the true objectness of ARC tasks.
    Section 4 and Table 5; the priors are generated using these abstractions, and the paper acknowledges abstraction selection can be wrong in Appendix A.9.
  • ad hoc to paper Training-set success is a valid criterion for selecting the abstraction and solution to evaluate on test.
    Algorithm 1 lines 22-24; the paper itself notes this can cause overfitting (A.9 and Figure 9).
  • ad hoc to paper Staged prompting reduces interference from irrelevant priors.
    Section 4; supported only by a limited ablation (A.6, Table 3) without GPT-o3-mini, and the effect is modest.
invented entities (3)
  • middle-vertical and middle-horizontal abstractions
    purpose: Split image into two equal components to support KAAR's objectness priors.
    New in KAAR (Table 5, red); no external validation or falsifiable handle outside the paper.
  • multi-lines abstraction
    purpose: Segment image by uniform-color rows or columns.
    New in KAAR; used for component decomposition, with no independent evidence beyond examples.
  • three-level ontology of priors
    purpose: Organize priors by dependency for staged augmentation.
    Section 4; qualitative structure with no independent test of the dependency ordering.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark." pith.science (2026). https://pith.science/paper/ZA4TQASZ

@misc{pith2026250517482,
  author       = {Pith},
  title        = {Pith review of: From Reasoning to Generalization: Knowledge-Augmented LLMs for ARC Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZA4TQASZ}},
  note         = {Machine review of arXiv:2505.17482}
}
read the original abstract

Recent reasoning-oriented LLMs have demonstrated strong performance on challenging tasks such as mathematics and science examinations. However, core cognitive faculties of human intelligence, such as abstract reasoning and generalization, remain underexplored. To address this, we evaluate recent reasoning-oriented LLMs on the Abstraction and Reasoning Corpus (ARC) benchmark, which explicitly demands both faculties. We formulate ARC as a program synthesis task and propose nine candidate solvers. Experimental results show that repeated-sampling planning-aided code generation (RSPC) achieves the highest test accuracy and demonstrates consistent generalization across most LLMs. To further improve performance, we introduce an ARC solver, Knowledge Augmentation for Abstract Reasoning (KAAR), which encodes core knowledge priors within an ontology that classifies priors into three hierarchical levels based on their dependencies. KAAR progressively expands LLM reasoning capacity by gradually augmenting priors at each level, and invokes RSPC to generate candidate solutions after each augmentation stage. This stage-wise reasoning reduces interference from irrelevant priors and improves LLM performance. Empirical results show that KAAR maintains strong generalization and consistently outperforms non-augmented RSPC across all evaluated LLMs, achieving around 5% absolute gains and up to 64.52% relative improvement. Despite these achievements, ARC remains a challenging benchmark for reasoning-oriented LLMs, highlighting future avenues of progress in LLMs.

Figures

Figures reproduced from arXiv: 2505.17482 by the authors.

Figure 1
Figure 1. An ARC problem example (25ff71a9) with image visualizations (a), including three input￾output pairs in the training instances, and one input image in the test instance, along with their corresponding 2D matrix representations (b). The ground-truth test output is enclosed in a red box. sorting, comparing quantities, performing basic arithmetic, and identifying numerical patterns; and (4) goal-directedness – inferring… view at source ↗
Figure 2
Figure 2. An illustration of the three ARC solution generation approaches, (1) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The example of goal-directedness priors augmentation in KAAR with input and response fragments from GPT-o3-mini. GPAR approaches goal-directedness priors by search￾ing for a sequence of program instructions [35] de￾fined in a DSL. Each instruction supports condition￾als, branching, looping, and action statements. KAAR incorporates the condition and action concepts from GPAR, and enables goal-directedness priors by a… view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Augmentation process in KAAR (block (b)) and the corresponding knowledge augmentation [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Asymmetric relative coverage matrices for RSPC [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Accuracy on test instances [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Accuracy on test instances It for RSPC and KAAR across average image size intervals, evaluated using GPT-o3-mini and QwQ-32B. See [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Variance in accuracy on Ir&It with in￾creasing iterations for RSPC and KAAR using GPT-o3-mini and QwQ-32B. See [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Fragment of ARC problem e7dd8335. Solution Analysis. RSPC achieves over 30% accuracy across evaluated metrics using GPT-o3-mini, even without knowledge augmentation. To assess its alignment with core knowledge priors, we manually reviewed RSPC-generated solution plans …
Figure 10
Figure 10. Figure 10: ARC problem 0520fde7 used in KAAR, organized by their prioritization. KAAR incorporates fundamental abstractions, such as 4-connected and 8-connected components, from GPAR, and extends them with additional abstractions unique to KAAR, highlighted in red [PITH_FULL_IM…
Figure 11
Figure 11. Figure 11: illustrates example ARC tasks for the movement, extension, recolor, and others categories in the public evaluation set. In the movement example, components are shifted to the image boundary in directions determined by their colors. The extension example is more comple…
Figure 12
Figure 12. Figure 12: Accuracy on test instances It for RSPC and KAAR across average image size intervals, evaluated with Gemini-2.0 and DeepSeek-R1-70B. 1 4 8 12 # Iterations 0 5 10 15 20 25 A c c u r a c y o n I &r It (%) Objectness Geometry, Topology, Numbers and Counting Goal-directedn…
Figure 13
Figure 13. Figure 13: Variance in accuracy on Ir&It with increasing iterations for RSPC and KAAR using Gemini-2.0 and DeepSeek-R1-70B. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Average token cost for augmenting priors at each level across four LLMs. K is [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: ARC problem 695367ec, where RSPC and KAAR generate the same code solution that passes the training instances but fails on the test instance using GPT-o3-mini. def generate_output_image(input_image): # Determine the border pattern based on the top row of the 6x6 input.…
Figure 16
Figure 16. Figure 16: ARC problem b1fc8b8e, where RSPC and KAAR generate the same code solution that passes the training instances but fails on the test instance using GPT-o3-mini. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Asymmetric relative coverage matrix of nine ARC solvers using GPT-o3-mini, showing [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Asymmetric relative coverage matrix of nine ARC solvers using Gemini-2.0, showing the [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: Asymmetric relative coverage matrix of nine ARC solvers using QwQ-32B, showing the [PITH_FULL_IMAGE:figures/full_fig_p028_19.png]
Figure 20
Figure 20. Figure 20: Asymmetric relative coverage matrix of nine ARC solvers using DeepSeek-R1-70B, [PITH_FULL_IMAGE:figures/full_fig_p029_20.png]
Figure 21
Figure 21. Figure 21: ARC problem 358ba94e, where repeated sampling with standalone code generation produces an incorrect solution using GPT-o3-mini. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_21.png]
Figure 22
Figure 22. Figure 22: ARC problem 15696249, where repeated sampling with standalone code generation produces an incorrect solution, whereas repeated sampling with planning-aided code generation yields the correct solution using GPT-o3-mini. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_22.png]
Figure 23
Figure 23. Figure 23: ARC problem d19f7514, where repeated sampling with planning-aided code generation produces a correct solution, whereas its refinement variant fails to refine the initial erroneous code, and the incorrect logic persists across subsequent refinements when using GPT-o3-m…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

79 extracted references · 64 canonical work pages

  1. [1]

    Machine learning in computer vision: A review.EAI Endorsed Transactions on Scalable Information Systems, 8 (32), 2021

    Abdullah Ayub Khan, Asif Ali Laghari, and Shafique Ahmed Awan. Machine learning in computer vision: A review.EAI Endorsed Transactions on Scalable Information Systems, 8 (32), 2021

    work page 2021

  2. [2]

    A survey of the usages of deep learning for natural language processing.IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020

    Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing.IEEE transactions on neural networks and learning systems, 32(2):604–624, 2020

    work page 2020

  3. [3]

    A survey of deep learning techniques for autonomous driving.Journal of field robotics, 37(3):362–386, 2020

    Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving.Journal of field robotics, 37(3):362–386, 2020

    2020

  4. [4]

    Building machines that learn and think like people.Behavioral and brain sciences, 40:e253, 2017

    Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people.Behavioral and brain sciences, 40:e253, 2017

    2017

  5. [5]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    arXiv 1911

  6. [6]

    Questions concerning certain faculties claimed for man.The Journal of Speculative Philosophy, 2(2):103–114, 1868

    Charles S Peirce. Questions concerning certain faculties claimed for man.The Journal of Speculative Philosophy, 2(2):103–114, 1868

    work page

  7. [7]

    Core knowledge.Developmental science, 10(1): 89–96, 2007

    Elizabeth S Spelke and Katherine D Kinzler. Core knowledge.Developmental science, 10(1): 89–96, 2007

    work page 2007

  8. [8]

    Program synthesis.Foundations and Trends® in Programming Languages, 4:1–119, 2017

    Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, et al. Program synthesis.Foundations and Trends® in Programming Languages, 4:1–119, 2017

    work page 2017

Show all 79 references
  1. [9]

    Graphs, constraints, and search for the abstraction and reasoning corpus

    Yudong Xu, Elias B Khalil, and Scott Sanner. Graphs, constraints, and search for the abstraction and reasoning corpus. InProceedings of the 37th AAAI Conference on Artificial Intelligence, AAAI, pages 4115–4122, 2023

    2023

  2. [10]

    Generalized planning for the abstraction and reasoning corpus

    Chao Lei, Nir Lipovetzky, and Krista A Ehinger. Generalized planning for the abstraction and reasoning corpus. InProceedings of the 38th AAAI Conference on Artificial Intelligence, AAAI, pages 20168–20175, 2024

    2024

  3. [11]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th Advances in Neural Information Processing Systems, NeurIPS, pages 24824–24837, 2022

    2022

  4. [12]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  5. [13]

    Competition-level code generation with alphacode.Science, 378:1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378:1092–1097, 2022

    2022

  6. [14]

    Codet: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. InProceedings of the 11th International Conference on Learning Representations, ICLR, pages 1–19, 2023

    2023

  7. [15]

    Coder reviewer reranking for code generation

    Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. Coder reviewer reranking for code generation. InProceedings of the 40th International Conference on Machine Learning, ICML, pages 41832–41846, 2023

    2023

  8. [16]

    Lever: Learning to verify language-to-code generation with execution

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. Lever: Learning to verify language-to-code generation with execution. In Proceedings of the 40th International Conference on Machine Learning, ICML, pages 26106– 26128, 2023

    2023

  9. [17]

    Debug like a human: A large language model debugger via verifying runtime execution step by step

    Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step by step. InFindings of the Association for Computational Linguistics: ACL 2024, pages 851–870, 2024. 10

    2024

  10. [18]

    Planning-driven program- ming: A large language model programming workflow.arXiv preprint arXiv:2411.14503, 2024

    Chao Lei, Yanchuan Chang, Nir Lipovetzky, and Krista A Ehinger. Planning-driven program- ming: A large language model programming workflow.arXiv preprint arXiv:2411.14503, 2024

    2024 arXiv

  11. [19]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. InProceedings of the 12th International Conference on Learning Representations, ICLR, 2024

    2024

  12. [20]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  13. [21]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InProceedings of the 13th International Conferenc...

    2025

  14. [22]

    Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–28, 2023

    Xue Jiang, Yihong Dong, Lecheng Wang, Fang Zheng, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models.ACM Transactions on Software Engineering and Methodology, 33(7):1–28, 2023

    2023

  15. [23]

    Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez

    Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. MapCoder: Multi-agent code generation for competitive problem solving. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL, pages 4912–4944, 2024

    2024

  16. [24]

    Evaluation of openai o1: Opportunities and challenges of agi.arXiv preprint arXiv:2409.18486, 2024

    Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, et al. Evaluation of openai o1: Opportunities and challenges of agi.arXiv preprint arXiv:2409.18486, 2024

    2024

  17. [25]

    Openai o3-mini.OpenAI, 2025

    OpenAI. Openai o3-mini.OpenAI, 2025. URL https://openai.com/index/openai-o 3-mini/. Accessed: 2025-03-22

    2025

  18. [26]

    Gemini 2.0 flash thinking.Google DeepMind, 2024

    Google DeepMind. Gemini 2.0 flash thinking.Google DeepMind, 2024. URL https: //deepmind.google/technologies/gemini/flash-thinking/. Accessed: 2025-03-22

    2024

  19. [27]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  20. [28]

    Alibaba cloud unveils qwq-32b: A compact reasoning model with cutting-edge performance.Alibaba Cloud, 2025

    Alibaba Cloud. Alibaba cloud unveils qwq-32b: A compact reasoning model with cutting-edge performance.Alibaba Cloud, 2025. URL https://www.alibabacloud.com/blog/alibab a-cloud-unveils-qwq-32b-a-compact-reasoning-model-with-cutting-edge-p erformance_602039. Accessed: 2025-03-22

    2025

  21. [29]

    Piaget’s cognitive developmen- tal theory: Critical review.Education Quarterly Reviews, 2(3):517–524, 2019

    Zana H Babakr, Pakstan Mohamedamin, and Karwan Kakamad. Piaget’s cognitive developmen- tal theory: Critical review.Education Quarterly Reviews, 2(3):517–524, 2019

    2019

  22. [30]

    Can llm be a good path planner based on prompt engineering? mitigating the hallucination for path planning.arXiv preprint arXiv:2408.13184, 2024

    Hourui Deng, Hongjie Zhang, Jie Ou, and Chaosheng Feng. Can llm be a good path planner based on prompt engineering? mitigating the hallucination for path planning.arXiv preprint arXiv:2408.13184, 2024

    2024 arXiv

  23. [31]

    LLM-a*: Large language model enhanced incremental heuristic search on path planning

    Silin Meng, Yiwei Wang, Cheng-Fu Yang, Nanyun Peng, and Kai-Wei Chang. LLM-a*: Large language model enhanced incremental heuristic search on path planning. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1087–1102, 2024

    2024

  24. [32]

    Large language models for mathematical reasoning: Progresses and challenges

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research ...

    2024

  25. [33]

    Contextual object detection with multimodal large language models.International Journal of Computer Vision, 133(2):825–843, 2025

    Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models.International Journal of Computer Vision, 133(2):825–843, 2025. 11

    2025

  26. [34]

    Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations.arXiv preprint arXiv:2305.18354, 2023

    Yudong Xu, Wenhao Li, Pashootan Vaezipoor, Scott Sanner, and Elias B Khalil. Llms and the abstraction and reasoning corpus: Successes, failures, and the importance of object-based representations.arXiv preprint arXiv:2305.18354, 2023

    2023 arXiv

  27. [35]

    Novelty and lifted helpful actions in generalized planning

    Chao Lei, Nir Lipovetzky, and Krista A Ehinger. Novelty and lifted helpful actions in generalized planning. InProceedings of the International Symposium on Combinatorial Search, SoCS, pages 148–152, 2023

    2023

  28. [36]

    Hypothesis search: Inductive reasoning with language models

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. Hypothesis search: Inductive reasoning with language models. InProceedings of the 12 th International Conference on Learning Representations, ICLR, 2024

    2024

  29. [37]

    H-arc: A robust estimate of human performance on the abstraction and reasoning corpus benchmark.arXiv preprint arXiv:2409.01374, 2024

    Solim LeGris, Wai Keen V ong, Brenden M Lake, and Todd M Gureckis. H-arc: A robust estimate of human performance on the abstraction and reasoning corpus benchmark.arXiv preprint arXiv:2409.01374, 2024

    2024 arXiv

  30. [38]

    Eval- uating spatial understanding of large language models.Transactions on Machine Learning Research, 2024

    Yutaro Yamada, Yihan Bao, Andrew Kyle Lampinen, Jungo Kasai, and Ilker Yildirim. Eval- uating spatial understanding of large language models.Transactions on Machine Learning Research, 2024

    2024

  31. [39]

    Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of llms.arXiv preprint arXiv:2304.11164, 2023

    Anthony G Cohn and Jose Hernandez-Orallo. Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of llms.arXiv preprint arXiv:2304.11164, 2023

    2023 arXiv

  32. [40]

    Do, Yan Xu, and Pascale Fung

    Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Love- nia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V . Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity...

    2023

  33. [41]

    Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects.arXiv preprint arXiv:2410.06405, 2024

    Wenhao Li, Yudong Xu, Scott Sanner, and Elias Boutros Khalil. Tackling the abstraction and reasoning corpus with vision transformers: the importance of 2d representation, positions, and objects.arXiv preprint arXiv:2410.06405, 2024

    arXiv

  34. [42]

    The raven’s progressive matrices: change and stability over culture and time

    John Raven. The raven’s progressive matrices: change and stability over culture and time. Cognitive psychology, 41(1):1–48, 2000

    2000

  35. [43]

    Measuring abstract reasoning in neural networks

    David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. InProceedings of the 37th International conference on machine learning, ICML, pages 511–520, 2018

    2018

  36. [44]

    Task cognition and planning for service robots.Intelligence & Robotics, (1):119–142, 2025

    Yongcheng Cui, Ying Zhang, Cui-Hua Zhang, and Simon X Yang. Task cognition and planning for service robots.Intelligence & Robotics, (1):119–142, 2025

    2025

  37. [45]

    From show to tell: A survey on deep learning-based image captioning.IEEE transactions on pattern analysis and machine intelligence, (1):539–559, 2022

    Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cucchiara. From show to tell: A survey on deep learning-based image captioning.IEEE transactions on pattern analysis and machine intelligence, (1):539–559, 2022

    2022

  38. [46]

    Visual question answering: from early developments to recent advances–a survey.arXiv preprint arXiv:2501.03939, 2025

    Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, and Hakim Hacid. Visual question answering: from early developments to recent advances–a survey.arXiv preprint arXiv:2501.03939, 2025

    2025 arXiv

  39. [47]

    Object detection with deep learning: A review.IEEE transactions on neural networks and learning systems, 30(11): 3212–3232, 2019

    Zhong-Qiu Zhao, Peng Zheng, Shou-tao Xu, and Xindong Wu. Object detection with deep learning: A review.IEEE transactions on neural networks and learning systems, 30(11): 3212–3232, 2019

    2019

  40. [48]

    Augmented language models: a survey.Transactions on Machine Learning Research, 2023

    Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey.Transactions on Mac...

    2023

  41. [49]

    KnowAgent: Knowledge-augmented planning for LLM-based agents

    Yuqi Zhu, Shuofei Qiao, Yixin Ou, Shumin Deng, Shiwei Lyu, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen, and Ningyu Zhang. KnowAgent: Knowledge-augmented planning for LLM-based agents. InFindings of the Association for Computational Linguistics: NAACL 2025, pages 3709–3732, 2025

    2025

  42. [50]

    FreshLLMs: Refreshing large language models with search engine augmentation

    Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. InFindings of the Association for Computational Linguistics: ACL ...

    2024

  43. [51]

    Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources

    Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. InProceedings of the 12th International Conference on Learning Re...

    2024

  44. [52]

    Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, ACL, pag...

    2023

  45. [53]

    Making language models better tool learners with execution feedback

    Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2024

  46. [54]

    1st place solution + code and official documentation

    J S Wind. 1st place solution + code and official documentation. https://www.kaggle.c om/competitions/abstraction-and-reasoning-challenge/discussion/154597 ,

  47. [55]

    Abstract visual reasoning enabled by language.arXiv preprint arXiv:2306.04091, 2023

    Giacomo Camposampiero, Loic Houmard, Benjamin Estermann, Joël Mathys, and Roger Wattenhofer. Abstract visual reasoning enabled by language.arXiv preprint arXiv:2306.04091, 2023

    2023 arXiv

  48. [56]

    An approach to solving the abstraction and reasoning corpus (arc) challenge.arXiv preprint arXiv:2306.03553, 2023

    Tan John Chong Min. An approach to solving the abstraction and reasoning corpus (arc) challenge.arXiv preprint arXiv:2306.03553, 2023

    2023 arXiv

  49. [57]

    Llms as a system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (arc) challenge

    John Chong Min Tan and Mehul Motani. Llms as a system of multiple expert agents: An approach to solve the abstraction and reasoning corpus (arc) challenge. InProceedings of the 2024 IEEE Conference on Artificial Intelligence, CAI, pages 782–787, 2024

    2024

  50. [58]

    Reflection system for the abstraction and reasoning corpus

    Kiril Bikov, Mikel Bober-Irizar, and Soumya Banerjee. Reflection system for the abstraction and reasoning corpus. InProceedings of the 2nd AI4Research Workshop: Towards a Knowledge- grounded Scientific Research Lifecycle, 2024

    2024

  51. [59]

    The llm architect: Solving arc-agi is a matter of perspective

    Daniel Franzen, Jan Disselhoff, and David Hartmann. The llm architect: Solving arc-agi is a matter of perspective. https://github.com/da-fr/arc-prize-2024/blob/main/the_ architects.pdf, 2024. Accessed: 2025-03-22

    2024

  52. [60]

    Addressing the abstraction and reasoning corpus via procedural example generation.arXiv preprint arXiv:2404.07353, 2024

    Michael Hodel. Addressing the abstraction and reasoning corpus via procedural example generation.arXiv preprint arXiv:2404.07353, 2024

    2024 arXiv

  53. [61]

    The conceptarc bench- mark: Evaluating understanding and generalization in the arc domain.arXiv preprint arXiv:2305.07141, 2023

    Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. The conceptarc bench- mark: Evaluating understanding and generalization in the arc domain.arXiv preprint arXiv:2305.07141, 2023

    2023 arXiv

  54. [62]

    Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis

    Wen-Ding Li, Keya Hu, Carter Larsen, Yuqing Wu, Simon Alford, Caleb Woo, Spencer M. Dunn, Hao Tang, Wei-Long Zheng, Yewen Pu, and Kevin Ellis. Combining induction and transduction for abstract reasoning. InProceedings of the 13th International Conference on Learning Representa...

    2025

  55. [63]

    Hysynth: Context-free llm approximation for guiding program synthesis

    Shraddha Barke, Emmanuel Anaya Gonzalez, Saketh Ram Kasibatla, Taylor Berg-Kirkpatrick, and Nadia Polikarpova. Hysynth: Context-free llm approximation for guiding program synthesis. InProceedings of the 38th Advances in Neural Information Processing Systems, NeurIPS, pages 156...

    2024

  56. [65]

    –” denotes no priors, while “ALL

    trained a probabilistic context-free grammar (PCFG) using LLM-generated plausible solutions to learn weighted functions. This enables the synthesizer to efficiently generate final program solutions. However, this approach requires a dedicated synthesizer for each DSL, limiting...

  57. [66]

    color change: color change involves modifying the value of a component, and the component size and position always does not change

  58. [67]

    movement: movement involves shifting the position of a component to a new location within the image, and the component size always does not change

  59. [68]

    extension: extending involves expanding the boundaries of a component to increase its size or reach within the image, and the component size always changes

  60. [69]

    completing: completing an image involves filling in missing or incomplete parts of a component to achieve a coherent and fully formed image

  61. [70]

    resizing: resizing involves altering the dimensions of a component by expanding or shrinking its size within the image

  62. [71]

    selecting: selecting involves identifying and isolating a specific component within the image as the output component, and the component size and color always does not change

  63. [72]

    copying: copying involves duplicating a component and either placing the duplicate in a new location or replacing the existing component within the image

  64. [73]

    flipping: flipping involves mirroring a component along a specified axis to reverse its orientation within the image. 40

  65. [74]

    rotation: rotation involves turning a component around a fixed point or center by a specified angle within the image

  66. [75]

    color change

    cropping: cropping involves cutting out a specific portion of a component. Please select the best suitable one or multiple categories from the provided list that best describe the task. Format your response by starting with [start category] and ending with [ end category], num...

  67. [76]

    selecting

    color chang [end category] If the task belongs to both "selecting" and "extension", your response should be: [start category]

  68. [77]

    ================================= User ================================= If this task involves ${action}:

    extension [end category] Prompt 13: Goal-directedness priors augmentation - target component idetification ================================ System ================================ You are an expert in analyzing grid-based image tasks, specifically in ${ action} components. ===...

  69. [78]

    - Refer to these components as TARGET components (e.g., component 1 in the first input image, component 2 and component 3 in the second input image, etc.)

    Begin by identifying WHICH COMPONENTS are to be ${action} in all input images (training and test pairs). - Refer to these components as TARGET components (e.g., component 1 in the first input image, component 2 and component 3 in the second input image, etc.). - List ALL targe...

  70. [79]

    - These conditions must be based on common priorities across all targeted components and must differ from the unselected components

    Determine the CONDITIONS used to select these TARGET components for ${ action} from each training and test input image. - These conditions must be based on common priorities across all targeted components and must differ from the unselected components. - For example: the size ...

  71. [2020]

    Accessed: 2025-03-22

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.