Pith. sign in

REVIEW 3 major objections 7 minor 25 references

Agent harnesses alone can lift a general open-weight model by ~52 points on ARC-AGI-1, under a dollar per task and with no ARC fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 21:48 UTC pith:GUJPT42X

load-bearing objection Solid agent-harness paper: controlled ~52-point lift on ARC public eval under a real budget, with a generation-bound diagnostic that the orchestrator is built to test; public-set and single-run caveats are real but do not erase the within-model evidence. the 3 major comments →

arxiv 2607.06764 v1 pith:GUJPT42X submitted 2026-07-07 cs.AI

Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

Kabir Moghe , Peter Chin This is my paper
classification cs.AI
keywords ARC-AGI-1agent harnessestest-time computeprogram synthesispattern explorationadaptive re-explorationpass@kcost-accuracy frontier
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a large share of ARC-AGI-1 performance is available through architecture rather than through expensive test-time search over frontier models or through training small models on ARC data. On the public 400-task set, a one-shot call to DeepSeek V3.2 in non-thinking mode scores 15.5%. A structured chain-of-thought call reaches 30%. An Explorer-Definer Pipeline that separates pattern discovery from executable program synthesis reaches 57.5% pass@2 at $0.25 per task. A Reflective Orchestrator that can spawn fresh exploration mid-loop when a hypothesis fails training pairs reaches 67.25% pass@2 at $0.62. Unbiased pass@k analysis shows the pipeline is generation-bound: selection already recovers about 95% of the candidate ceiling, so further gains require broader generation. The orchestrator implements that prediction and confirms it with nearly equal lifts on naive pass@2 and unbiased pass@1. A sympathetic reader cares because the result claims that careful agent design recovers most of the gap between cheap single-call baselines and expensive leaderboard systems without specializing the model to the benchmark.

Core claim

Three architectural levers—within-call deliberation, across-call decomposition of pattern discovery from transformation synthesis, and adaptive re-exploration when a hypothesis fails—lift the same non-thinking open-weight model from 15.5% one-shot to 67.25% pass@2 on the ARC-AGI-1 public set at under a dollar per task, with no ARC-specific fine-tuning. The pipeline is generation-bound rather than selection-bound, and the orchestrator’s near-equal lifts on selection-free and selection-mediated metrics confirm that adaptive re-exploration supplies the missing generation diversity.

What carries the argument

The Explorer-Definer Pipeline (parallel PatternExplorers producing compressed natural-language findings, then TransformationDefiners that emit and refine executable Python scored on training pairs) plus the Reflective Orchestrator’s explore_new_patterns tool, which mid-loop spawns focused explorers under negative guidance when the current frame fails.

Load-bearing premise

That the absolute scores and the size of the architectural lifts measured on the public 400-task set, with the main configuration chosen by inspecting that same set, will transfer to the semi-private holdout and are not inflated by public-set artifacts or residual ARC exposure in the model’s general post-training.

What would settle it

Run the identical harnesses, prompts, and scoring code on the official semi-private ARC-AGI-1 holdout (or a true held-out split) with the same model and budget; if the ~52-point total lift or the pipeline-to-orchestrator generation-side lift collapses or shrinks dramatically, the central claim about transferable architectural recovery fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A meaningful fraction of ARC-AGI-1 accuracy is recoverable on open-weight models by agent design alone, without ARC fine-tuning or multi-dollar test-time search.
  • When unbiased pass@k shows selection already near the candidate ceiling, further budget should be spent on broader generation (more explorers or mid-loop re-exploration) rather than better ranking.
  • A private think scratchpad is a measurable load-bearing component of the pipeline; removing it costs roughly 6 pass@2 points.
  • The same three-lever progression and generation-bound diagnosis can be used as a design loop for other few-shot abstract-reasoning settings that supply training-pair verifiers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If generation-bound failures dominate, tools that let synthesis stages request fresh upstream abstractions under negative guidance may transfer to other program-induction or puzzle domains where multiple rules fit the examples.
  • Dedup-by-test-prediction plus mid-loop spawn together act as an implicit diverse-voting scheme; similar selection-plus-re-exploration pairs could be tested on other multi-hypothesis agent systems.
  • A controlled spawn-off ablation and full-scale multi-model panel would convert the correlational evidence that spawn drives unique solves into tighter causal and generality claims.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper studies a third regime for ARC-AGI-1: an open-weight model (DeepSeek V3.2, non-thinking) under a strict budget with no ARC-specific fine-tuning, asking how much accuracy is recoverable through agent harness architecture alone. It introduces an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis (57.50% pass@2 at $0.25/task) and a Reflective Orchestrator that adds mid-loop adaptive re-exploration when hypotheses fail on training pairs (67.25% pass@2 at $0.62/task), lifting a 15.50% one-shot baseline by ~52 points. A generation-bound diagnostic (unbiased pass@k; train-score selection recovers ~95% of the candidate ceiling) motivates the orchestrator; near-equal lifts on naive pass@2 (+9.75 pp) and unbiased pass@1 (+9.81 pp) are presented as confirmation that gains are generation-side. Controlled ablations (think tool −5.75 pp; refinement −0.75 pp), an N×t×M Pareto surface, paired bootstrap CIs, and a 99-task Qwen3-235B cross-model check support the architectural claims.

Significance. If the within-model lifts hold, the work occupies a sparsely reported sub-$1/task region of the ARC cost–accuracy plane and shows that decomposition, a think scratchpad, and adaptive re-exploration can recover a large fraction of the gap between cheap one-shot calls and heavy test-time or ARC-specialized systems without benchmark-specific training. Strengths include: (i) tightly controlled within-model comparisons with token-normalized costs and paired bootstrap deltas; (ii) a falsifiable generation-vs-selection diagnostic that is then architecturally tested; (iii) load-bearing ablations (especially the think tool); (iv) public code and prompts; and (v) honest public-set and configuration-selection caveats. The diagnostic-then-test framing is a useful methodological contribution for agentic systems papers more generally.

major comments (3)
  1. [§4.7 Generation-bound diagnostic] §4.7 (and Abstract): The generation-bound claim and the orchestrator confirmation rest on a single unreplicated M=5 candidate pool per architecture. Unbiased pass@1 (46.40% pipeline; 56.21% orchestrator), the ~95% ceiling-recovery figure, and the near-equality of +9.75 pp naive pass@2 vs +9.81 pp unbiased pass@1 are all computed from one draw. Given the strongly bimodal correct-candidate counts and the paper’s own ~±2 pp single-run noise floor (§6), modest run-to-run movement of tasks into/out of the ‘none correct’ bin can change both the ceiling fraction and the lift equality by several points. This is load-bearing for Contribution 2 and for the claim that the orchestrator ‘confirms’ a generation-side prediction. Please either (a) report multi-seed M=5 pools (or a substantial subset) with uncertainty on unbiased pass@k and ceiling recovery, or (b) substantially soften the confirmation l
  2. [§4.8 Behavioral characterization] §4.8 / §7: Spawn is presented as the dominant mechanism behind the orchestrator lift (69–75% of unique solves use explore_new_patterns), but there is no spawn-off ablation. The correlational task analysis and Appendix A case study are informative, yet they do not isolate spawn from the rest of the agentic loop (done exit, longer iteration budget, different prompting). Because the paper’s design narrative is that spawn is the relief valve for wrong-abstraction failures, a controlled spawn-off run (even on a fixed subset) is needed to make that causal claim, or the text should consistently frame the 75% figure as correlational evidence only and move stronger causal language to future work.
  3. [§6 Limitations; §4.4 Pareto surface] §6 Configuration selection: The canonical (N=5, t=0.5, M=5) point was chosen by inspecting the Pareto surface on the same 400-task public evaluation set used for all headline comparisons. The three mitigations offered (smooth surface; deltas hold across much of the surface; Qwen3 ordering) reduce but do not remove selection bias on absolute levels and on which cell is starred. At minimum, report the full surface numbers for nearby cells as primary results (not only the starred cell), and state clearly that absolute pass@2 at the chosen operating point is not an independent holdout estimate. Preferable: re-select on a validation split or pre-register the cell and re-evaluate the remainder.
minor comments (7)
  1. [Abstract / §1] Public-evaluation caveat is well written but should appear earlier in the abstract or as a footnote on the headline numbers so readers do not treat 57.50%/67.25% as leaderboard-comparable without reading §6.
  2. [Table 1] Table 1 reports pipeline pass@1 and pass@2 but only orchestrator pass@2. For symmetry and for the unbiased-pass@1 discussion in §4.7, include orchestrator pass@1 (or state why it is omitted).
  3. [§3.4 Train-feedback refinement] The 50% train-accuracy refinement threshold (§3.4) is described as probe-chosen and not rigorously tuned; a one-sentence sensitivity note (e.g., 40%/60%) would help readers judge robustness of the small refinement ablation.
  4. [§3.1 Cost accounting; Figures 4–7] Figure 4/5 cost axes and AtlasCloud FP8 rates are clear; please also report mean tokens (prompt/completion) per architecture so costs remain interpretable if list prices change.
  5. [§4.9 / Table 4] Cross-model Qwen3 result (n=99, M=3) is valuable; the CI for Qwen3 pipeline→orchestrator touches zero—state more explicitly in the main text that this is underpowered rather than evidence of a weaker lift.
  6. [References] Minor: arXiv ID and some reference years in the provided text look placeholder-like (e.g., 2026 dates); verify bibliography consistency before camera-ready.
  7. [Appendix A; §4.8] Appendix A task 3a301edc is excellent for intuition; a short pointer in §4.8 to the figure/table numbers would help readers find it from the main text.

Circularity Check

0 steps flagged

No circularity: empirical harness results on external ARC tasks; generation-bound diagnostic and orchestrator confirmation are measurements, not tautologies.

full rationale

This is a self-contained empirical systems paper. Headline accuracies (pipeline 57.50% pass@2, orchestrator 67.25% pass@2) and costs are measured by exact-match scoring on held-out ARC-AGI-1 public test grids and by token counts at published list rates—not derived by redefining the target. The generation-bound claim applies the standard unbiased Codex pass@k estimator to held M=5 candidate pools and reports that train-score selection recovers ~95% of the pass@M ceiling; that is a property of the observed candidate distribution, not a fit renamed as prediction. The orchestrator is then a design response whose lift is re-measured on both naive pass@2 (+9.75 pp) and unbiased pass@1 (+9.81 pp); near-equality is an empirical comparison of two estimators on two systems, not forced by construction. Selection via training-pair accuracy uses the few-shot pairs that are part of the ARC task definition, with test grids held out—standard and non-circular. Configuration choice on the same public set and single-run candidate pools are validity/robustness concerns, not circular reductions. No self-definitional equations, no fitted-input-as-prediction, no load-bearing self-citation uniqueness chain, and no renamed known result. Score 0 with empty steps is the correct finding.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

Load-bearing content is empirical architecture plus evaluation protocol, not a formal derivation. Free parameters are harness hyperparameters chosen or lightly probed. Axioms are standard ARC evaluation assumptions and the modeling choice that train-pair exact match is a valid selection/refinement signal. Invented entities are the two agent architectures and the generation-bound diagnostic framing—method constructs with evidence only inside this study’s runs.

free parameters (6)
  • canonical explorer count N
    N∈{1,2,5} swept; N=5 chosen as dominant cost–accuracy lever and headline operating point after inspecting the Pareto surface on the evaluation set.
  • explorer temperature t
    t∈{0.0,0.5,1.0} swept; t=0.5 selected as intermediate temperature that dominates the frontier at N=5.
  • definer best-of M
    M∈{1..5}; M=5 used for headline pipeline/orchestrator despite diminishing returns past M=3.
  • spawn fan-out K and MAX_SPAWN_CALLS
    K=2 explorers per spawn and max 3 spawns chosen from pipeline heuristics (N=1→2 marginal gain; cost control), not orchestrator-specific sweep.
  • refinement training-accuracy threshold (50%)
    Refine when train accuracy is at least half but not all; set via informal probes, not rigorously tuned.
  • orchestrator fuses (MAX_ITERATIONS=15, MAX_CONSECUTIVE_EXEC_ERRORS=5)
    Hard caps that shape cost and exit-mode distribution; MAX_ITERATIONS tail is a major cost driver.
axioms (5)
  • domain assumption Exact cell-level match on ARC training pairs is a valid verifier and ranking signal for candidate programs (including dedup-by-test-prediction).
    Core selection and refinement mechanism throughout §3.4–3.5; standard for ARC program-synthesis systems but still an assumption about transfer to test grids.
  • domain assumption General-purpose post-training of DeepSeek V3.2 (including RL components) can be treated as fixed background so architectural deltas are not confounded by ARC-specific training, aside from possible public leakage.
    Stated in Introduction “Contextualizing the no RL or fine-tuning condition”; absolute levels still depend on that post-training.
  • domain assumption Public 400-task evaluation set is an acceptable proxy for controlled architectural comparison even though it is not the semi-private leaderboard set.
    Evaluation protocol §3.1 and Limitations §6; authors flag possible overestimation.
  • standard math Unbiased pass@k on M=5 candidate sets diagnoses generation vs selection bottlenecks for these harnesses.
    Uses Chen et al. estimator; interpretation that ~95% ceiling recovery implies generation-bound is a modeling judgment applied in §4.7.
  • ad hoc to paper Negative-leaning guidance to spawned explorers is higher-information than open-ended re-exploration after failed hypotheses.
    Design choice in §3.5 motivated information-theoretically and illustrated in Appendix A, not independently validated by ablation.
invented entities (3)
  • Explorer-Definer Pipeline no independent evidence
    purpose: Two-stage agent harness separating parallel pattern exploration from executable transformation synthesis with train-pair verification.
    Primary architecture; evidence is internal experimental performance and ablations only.
  • Reflective Orchestrator no independent evidence
    purpose: Agentic synthesis loop with explore_new_patterns spawn tool to broaden generation when hypotheses fail.
    Architectural test of the generation-bound diagnostic; support is within-paper lifts and trace statistics.
  • Generation-bound (vs selection-bound) diagnostic for ARC harnesses no independent evidence
    purpose: Falsifiable claim that train-score selection already captures ~95% of candidate ceiling, so accuracy needs broader generation.
    Methodological framing that motivates the orchestrator; confirmed only on these candidate distributions.

pith-pipeline@v1.1.0-grok45 · 27205 in / 3924 out tokens · 160286 ms · 2026-07-10T21:48:38.830954+00:00 · methodology

0 comments
read the original abstract

Recent progress on ARC-AGI-1 from disclosed architectures has come broadly from two regimes: heavy test-time compute over frontier models (evolutionary search, exhaustive sampling, extended chain-of-thought), or benchmark-specific training in which small models are fine-tuned on ARC data, often with task-specialized architectures. We study a third regime: an open-weight model in non-thinking mode (DeepSeek V3.2) under a strict budget, with no ARC-specific fine-tuning. We study what is recoverable through architecture alone, building agentic harnesses that decompose pattern-discovery and program-synthesis stages explicitly. First, we introduce an Explorer-Definer Pipeline that separates pattern discovery from executable transformation synthesis, implemented as a two-stage agent pipeline. Next, we present the Reflective Orchestrator, which augments the pipeline with autonomous exploration of new transformations when previous hypotheses fail on training pairs. On the ARC-AGI-1 public 400-task evaluation set, the pipeline reaches 57.50% pass@2 at \$0.25 per task, and the orchestrator reaches 67.25% pass@2 at \$0.62 per task. Together these architectures lift a 15.50% one-shot baseline by ~52 points without benchmark-specific training or heavy test-time compute. Furthermore, the orchestrator-driven lift tests a falsifiable diagnostic the pipeline produces; unbiased pass@k analysis suggests the pipeline is generation-bound, not selection-bound (selection via training-pair accuracy captures ~95% of the candidate ceiling) and predicts that significant improvement requires broader generation, not better ranking. The orchestrator implements this prediction via adaptive re-exploration and confirms it (unbiased pass@1 lift +9.81 pp, matching selection-mediated pass@2 lift). An additional pipeline ablation identifies its think tool as a significant component, with removal reducing pass@2 by 5.75 pp.

Figures

Figures reproduced from arXiv: 2607.06764 by Kabir Moghe, Peter Chin.

Figure 1
Figure 1. Figure 1: PatternExplorer agent. Explorers are given a generous per-agent token budget (∼100k tokens) and are instructed to be exhaustive: examine symmetries, color mappings, spatial transformations, object detection, counting regularities, and any other structural properties of the input–output pairs. At the end of exploration, each agent is forced to emit a synthesis output containing a structured markdown summary… view at source ↗
Figure 2
Figure 2. Figure 2: Explorer-Definer Pipeline with a single TransformationDefiner agent. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: TransformationDefiner as a Reflective Orchestrator. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: plots the 45 pipeline configurations on the cost–pass@2 plane, faceted by explorer tempera￾ture. Three structural facts emerge [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Explorer-Definer Pipeline and Reflective Orchestrator across [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Cross-model Pareto on a matched 99-task subset of ARC-AGI-1. DeepSeek V3.2 (filled [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Headline cost–accuracy frontier across the four primary architectures. DeepSeek V3.2 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Task 3a301edc, training pairs and test pair (input on top, expected output on bottom) [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 16 internal anchors

  1. [1]

    Blog post, Jeremy’s Substack

    Jeremy Berman.Getting 50% (SoTA) on ARC-AGI with GPT-4o. Blog post, Jeremy’s Substack. Evolutionary Test-Time Compute; evolves Python transform functions. Accessed 2026-05-19. 2024.URL: https://jeremyberman.substack.com/p/how- i- got- a- record- 536- on-arc-agi

    work page 2026

  2. [2]

    Blog post, Jeremy’s Substack

    Jeremy Berman.How I Got the Highest Score on ARC-AGI Again, Swapping Python for English. Blog post, Jeremy’s Substack. Evolutionary Test-Time Compute with natural-language instructions; 79.6% on ARC-AGI-1, 29.4% on ARC-AGI-2 with Grok-4. Accessed 2026-05-19. 2025.URL: https://jeremyberman.substack.com/p/how- i- got- the- highest- score-on-arc-agi-again

    work page 2026

  3. [3]

    Blog post, CT Pang’s Substack

    Eric Pang.Efficient Evolutionary Program Synthesis: How I Broke the Efficiency Frontier With a DreamCoder-Inspired Approach. Blog post, CT Pang’s Substack. DreamCoder-inspired LLM-assisted program synthesis; 77.1% on ARC-AGI-1, 26.0% on ARC-AGI-2, estimated $2.56 per task on ARC-AGI-1. Code: github.com/epang080516/arc_agi. Accessed 2026-05-

    work page 2026

  4. [4]

    2025.URL: https://ctpang.substack.com/p/arc- agi- 2- sota- efficient- evolutionary

    work page 2025

  5. [5]

    Blog post, Redwood Re- search

    Ryan Greenblatt.Getting 50% (SoTA) on ARC-AGI with GPT-4o. Blog post, Redwood Re- search. Program synthesis with extensive sampling. Accessed 2026-05-19. 2024.URL: https: //redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt

    work page 2026

  6. [6]

    Less is More: Recursive Reasoning with Tiny Networks

    Alexia Jolicoeur-Martineau.Less is More: Recursive Reasoning with Tiny Networks. Tiny Recursive Model (TRM); 7M parameters, 45% on ARC-AGI-1. 2025. arXiv: 2510.04871 [cs.LG].URL:https://arxiv.org/abs/2510.04871

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective

    Daniel Franzen, Jan Disselhoff, and David Hartmann.The LLM ARChitects: Solving ARC- AGI Is a Matter of Perspective. ARChitects competition entry; depth-first search with fine- tuned LLMs and test-time augmentation. 2025. arXiv: 2505.07859 [cs.LG].URL: https: //arxiv.org/abs/2505.07859

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  8. [8]

    François Chollet et al.ARC Prize 2024: Technical Report. 2024. arXiv:2412.04604 [cs.AI]. URL:https://arxiv.org/abs/2412.04604

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  9. [9]

    2025.URL: https://arcprize

    ARC Prize Foundation.ARC Prize Verified Testing Policy. 2025.URL: https://arcprize. org/policy(visited on 06/02/2026)

    work page 2025

  10. [10]

    Evaluating Large Language Models Trained on Code

    Mark Chen et al.Evaluating Large Language Models Trained on Code. Introduces the unbiased pass@k estimator. 2021. arXiv: 2107.03374 [cs.LG] .URL: https://arxiv.org/abs/ 2107.03374

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  11. [11]

    On the Measure of Intelligence

    François Chollet. “On the Measure of Intelligence”. In:arXiv preprint arXiv:1911.01547 (2019).URL:https://arxiv.org/abs/1911.01547

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  12. [12]

    François Chollet et al.ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems. 2025. arXiv:2505.11831 [cs.AI].URL:https://arxiv.org/abs/2505.11831

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  13. [13]

    Antonio Roye-Azar et al.Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute. 2025. arXiv: 2512.11847 [cs.LG] .URL: https: //arxiv.org/abs/2512.11847. 21

    work page arXiv 2025

  14. [14]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    Xuezhi Wang et al. “Self-Consistency Improves Chain of Thought Reasoning in Language Models”. In:International Conference on Learning Representations (ICLR). 2023. arXiv: 2203.11171 [cs.CL].URL:https://arxiv.org/abs/2203.11171

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Xinyun Chen et al.Universal Self-Consistency for Large Language Model Generation. 2023. arXiv:2311.17311 [cs.CL].URL:https://arxiv.org/abs/2311.17311

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Reflexion: Language Agents with Verbal Reinforcement Learning

    Noah Shinn et al. “Reflexion: Language Agents with Verbal Reinforcement Learning”. In: Advances in Neural Information Processing Systems (NeurIPS). 2023.URL: https://arxiv. org/abs/2303.11366

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models

    Andy Zhou et al. “Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models”. In:Proceedings of the 41st International Conference on Machine Learning (ICML). 2024.URL:https://arxiv.org/abs/2310.04406

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Self-Refine: Iterative Refinement with Self-Feedback

    Aman Madaan et al. “Self-Refine: Iterative Refinement with Self-Feedback”. In:Advances in Neural Information Processing Systems (NeurIPS). 2023. arXiv: 2303.17651 [cs.CL].URL: https://arxiv.org/abs/2303.17651

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Anthropic Engineering Blog

    Anthropic.The "think" tool: Enabling Claude to stop and think in complex tool use situ- ations. Anthropic Engineering Blog. Mar. 2025.URL: https://www.anthropic.com/ engineering/claude-think-tool(visited on 03/20/2025)

    work page 2025

  20. [20]

    DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning

    Kevin Ellis et al. “DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning”. In:Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI). 2021.URL: https://arxiv. org/abs/2006.08381

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  21. [21]

    Communicating Natural Programs to Humans and Machines

    Sam Acquaviva et al. “Communicating Natural Programs to Humans and Machines”. In: Advances in Neural Information Processing Systems (NeurIPS)(2022).URL: https : / / arxiv.org/abs/2106.07824

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  22. [22]

    Charlie Snell et al.Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. 2024. arXiv:2408.03314 [cs.LG].URL: https://arxiv.org/ abs/2408.03314

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Let's Verify Step by Step

    Hunter Lightman et al. “Let’s Verify Step by Step”. In:International Conference on Learning Representations (ICLR). 2024.URL:https://arxiv.org/abs/2305.20050

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  24. [24]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”. In:Advances in Neural Information Processing Systems (NeurIPS). 2022.URL: https : //arxiv.org/abs/2201.11903

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  25. [25]

    Scalable Best-of-N Selection for Large Language Models via Self-Certainty

    Zhewei Kang, Xuandong Zhao, and Dawn Song. “Scalable Best-of-N Selection for Large Language Models via Self-Certainty”. In:Advances in Neural Information Processing Systems (NeurIPS). 2025. arXiv: 2502.18581 [cs.CL] .URL: https://arxiv.org/abs/2502. 18581. A Canonical task:3a301edc This appendix walks through a single ARC-AGI-1 evaluation task on which th...

    work page arXiv 2025