Pith. sign in

REVIEW 5 major objections 6 minor 53 references

Generating Diverse Hypotheses for Inductive Reasoning

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By asking an LLM to first list distinct concepts and then generate one hypothesis per concept, this paper claims inductive reasoning improves by several accuracy points at equal compute.

desk verdict MoC's two-stage concept-then-hypothesis sampling is a simple, likely real gain over IID search, but the paper needs error bars and a direct check of its mechanism before the headline numbers are robust. read the letter →

arxiv 2412.13422 v2 pith:3C5FHNTV submitted 2024-12-18 cs.AI cs.SE

Kang-il Lee , Hyukhun Koh , Dongryeol Lee , Seunghyun Yoon , Minsung Kim , Kyomin Jung This is my paper · ORCID
classification cs.AIcs.SE
keywords MixtureofConceptsinductivereasoninghypothesisdiversityLLMsamplingprogramsynthesistextdegenerationprogrammingbyexampletemperature
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that when LLMs solve inductive-reasoning problems by sampling many candidate rules and keeping the one that fits the training examples, plain independent sampling wastes most of its budget because failed hypotheses are often repeats. It shows that raising the sampling temperature only helps up to a point, since high temperatures degrade the model's text into unparseable output. It then introduces Mixture of Concepts (MoC), a two-stage method that first asks the LLM to list K semantically distinct conceptual clues, then feeds each clue as a hint for generating one hypothesis. On four benchmarks spanning list transforms, grid transformations, Python programming, and string transformations, MoC raises test accuracy by roughly 3 to 5 percentage points over the independent baseline with the same number of hypotheses, and it solves some problems that resisted more than 500 baseline samples using only 64 hypotheses.

What carries the argument

Mixture of Concepts (MoC) is a two-stage procedure: a concept-proposal prompt instructs the LLM to output K elementary concepts in JSON form, and a hypothesis-generation prompt then uses each parsed concept as an explicit hint for writing a natural-language hypothesis and a Python function. The mechanism exploits the autoregressive property that sequential list generation tends to avoid repeating earlier items, yielding concepts that are 'rarely semantically redundant.' The concept hint then steers each generated hypothesis toward a different region of the rule space, so the same number of samples covers more ground than independent sampling.

What would settle it

Take the Table 2 problem sets and rerun MoC after replacing each proposed concept with an independent random concept drawn from a small fixed pool while keeping K fixed; if accuracy stays at the MoC level, the specific concept proposals are irrelevant, and if it drops to the baseline level, the concept-proposal stage is doing the work.

Watch

Extended reading notes

Core claim

The central claim is that the bottleneck in LLM inductive reasoning is not the absence of correct hypotheses but semantic redundancy in sampling: on failed List Functions instances, 32 generated responses contain on average only 7.89 unique programs. Temperature-based diversity saturates because high temperatures cause text degeneration, so the paper proposes MoC, in which an LLM first lists K 'elementary concepts' and then each concept is parsed and inserted as a hint into the hypothesis-generation prompt. This converts K draws from one concentrated distribution into K draws from K different conditioned distributions. The paper shows that the accuracy gain is attributable to diversity rather than better reasoning, because greedy MoC with a single hypothesis roughly matches the greedy baseline.

Load-bearing premise

The whole gain rests on the premise that prompting an LLM to list K concepts produces K genuinely distinct and useful concepts, and that feeding each concept as a hint actually steers hypothesis generation into a different part of the rule space; if concepts collapse or are ignored, MoC degenerates to ordinary sampling.

Editorial extensions

If this is right

  • At equal compute, MoC consistently outperforms independent sampling across GPT-4o-mini, GPT-4o, Llama-3.1-70B, and Qwen2.5-72B, with average gains of 3.3 to 5.0 percentage points at K=8.
  • MoC achieves roughly the same accuracy as the baseline while generating only half as many hypotheses, improving the compute efficiency of hypothesis search.
  • Challenging problems that resisted more than 500 independent samples were solved with 64 MoC-generated hypotheses, indicating that exploration strategy, not model capability, is often the limiting factor.
  • Balancing concepts and resamples matters: with a fixed total budget, C=8 concepts with S=2 hypotheses per concept outperforms both fewer concepts with more resamples and more concepts with one resample each.
  • Temperature is not a reliable diversity lever for this task because quality declines before diversity meaningfully increases, motivating structural diversity mechanisms like MoC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • MoC's concept list could be reused across related problems or ranked by past success, turning a per-problem prompt trick into a growing library of search priors, though the paper does not explore this.
  • The concept-conditioned sampling idea is not limited to program induction; any generate-and-test search over structured outputs, such as natural-language rule induction or constraint satisfaction, could adopt the same two-stage diversity mechanism.
  • The paper does not directly measure whether the proposed concepts are actually semantically distinct or whether the LLM follows the hint, so a cheap diagnostic would be to compare output-equivalence distances among concept-conditioned hypotheses and to test MoC under concept lists with controlled overlap.
  • The Limitations section notes that diversity amplification could also amplify social bias and toxicity in natural-language reasoning, an implicit caution that the same mechanism which surfaces rare correct rules could surface rare harmful content.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript addresses hypothesis diversity in LLM-based inductive reasoning, where an LLM is prompted to propose Python functions and the function consistent with training examples is tested on held-out cases. The authors first analyze IID sampling and temperature, finding that higher temperatures initially improve diversity and accuracy but eventually cause text degeneration and saturation. They then propose Mixture of Concepts (MoC): an LLM first proposes K elementary concepts, each concept is used as a hint to generate a hypothesis, and the best train-consistent hypothesis is evaluated on test examples. Experiments are reported on four inductive reasoning benchmarks (List Functions, MiniARC, MBPP+, Playgol-str) with four LLMs, claiming accuracy improvements over an IID baseline (e.g., +4.5pp for GPT-4o-mini and +5.0pp for Llama-3.1-70B at K=8), increased numbers of unique executable hypotheses, and case studies of hard problems solved with K=64 hypotheses.

Significance. If the reported gains are reproducible, MoC is a simple, parallelizable alternative to iterative refinement for LLM-based inductive reasoning, and the temperature-degeneration analysis is a useful practical caution. The evaluation has a genuine strength: hypotheses are executable Python programs and accuracy is measured on held-out test cases, so the correctness signal is objective and does not depend on fitted parameters. The K=1 greedy control is also a reasonable attempt to separate reasoning gains from diversity gains. The main reservations are statistical and mechanistic: the headline results are single-run estimates without variance, and the proposed concept-non-redundancy mechanism is asserted rather than directly measured. These concerns are fixable and do not invalidate the core idea, but they are load-bearing for the paper's claims.

major comments (5)
  1. [§4.2, Tables 2 and 3] The main accuracy results are single-point estimates with no error bars, no number of independent runs, and no significance tests. With only 100-150 test instances per dataset, the standard error of an accuracy near 40% is roughly 4-5 percentage points, so the reported +4.5pp and +5.0pp improvements are not distinguishably different from sampling noise. The abstract's claim that MoC 'demonstrated significant performance improvements' needs at least mean±std over multiple seeds or confidence intervals before it is supportable. The same issue applies to Tables 5, 7, and 8, which are used for scaling and configuration recommendations.
  2. [§3, Concept Proposal] The claim that 'when instructing an LLM to generate a list of items, the generated items are rarely semantically redundant' is backed only by 'empirical investigation' with no quantification. Table 3 measures the number of unique final Python programs, not the redundancy of the K concepts, and the paper does not measure how often the generated hypothesis actually follows the hint. Appendix D's example concept lists contain pairs such as 'matrix'/'grid' and 'numerical sequences'/'number theory' that appear semantically close. Without a direct measure of concept redundancy (e.g., embedding similarity or human annotation) and a hint-adherence check (e.g., comparing the generated hypothesis distribution with and without the hint), the diversity gain could be due to the extra instruction text or prompt format rather than to the proposed concept-conditioning mechanism.
  3. [§2.2, Figures 3 and 4] The temperature analysis is confounded by changing the sampling algorithm: for temperatures 1.67 and 2.0, the authors switch to top-p sampling with p=0.95, so the observed saturation cannot be attributed to temperature alone. The analysis also uses a single model (GPT-4o-mini) and no error bars, yet §4.2 justifies the choice T=1.0 for all models based on this result. To support the temperature claim, the experiments should either keep the sampling scheme fixed or explicitly ablate top-p at each temperature, and should include variance information.
  4. [§4.2, Table 2 and Appendix C] The comparison with Iterative Hypothesis Refinement (IHR) is incomplete: IHR is evaluated only on List Functions and MiniARC and only with GPT-4o-mini and GPT-4o, while MoC is tested on all four benchmarks with four models. The IHR configuration T=2, N=4 generates 8 hypotheses but also uses refinement, so the compute comparison should be made explicit. Without the full benchmark/model matrix, the claim that MoC outperforms 'other approaches' beyond the IID baseline is not fully supported. The Appendix C comparison with Hypothesis Search is also based on a value read from a figure in a previous paper and should be replaced with exact numbers or an identical rerun.
  5. [§4.3, Table 5] The conclusion that C=8, S=2 is the best allocation of the hypothesis budget is based on single-run test-set comparisons with no variance. Because the best configuration is selected after observing test accuracy, this is susceptible to selection bias. The paper should report variance over independent runs or use a separate validation set to select C and S before reporting test accuracy.
minor comments (6)
  1. [§2.1] The definition of semantically unique hypotheses depends on executing programs on an unspecified set of inputs. Please specify the exact input set used for equivalence testing (e.g., all lists within certain length/value bounds, or only the train and test inputs), since it affects the uniqueness counts in Tables 1 and 3.
  2. [§3 and Appendix A] The MoC concept-proposal prompt for GPT-4o adds the instruction 'The concepts should be diverse, simple and concise,' which is not used for the other models. This is a confound when comparing models; please justify the difference or use identical prompts.
  3. [§4.3, Table 6] The phrase 'over 500 IID samples' is vague. Please report the exact number of baseline samples and the total compute cost of MoC, including the concept-proposal tokens, to support the efficiency claim.
  4. [Figure 4] The x-axis label 'Num. Samples (Power of 2)' is misleading because the axis appears to show the actual number of samples (1, 2, 4, 8, 16, 32), not a power-of-two index. Please rename the axis or change the tick labels.
  5. [Appendix C] The comparison with Hypothesis Search relies on a value ('appears to be 59') read from a graph in the prior paper. This is not a rigorous numerical comparison; either rerun the baseline under identical conditions or report the exact number from the original source.
  6. [Various] There are minor typos: 'lanuguages' in §2.1, 'morhological operations' in Appendix D, and 'abalation' if it remains in the revised text. These should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoC's accuracy and diversity claims are validated against held-out test cases with executable Python, and no fitted parameter is renamed as a prediction.

full rationale

The paper's derivation chain is empirical rather than definitional. The baseline samples K hypotheses as Python functions and submits the first function matching all train examples; MoC changes only the sampling procedure by first proposing K concepts and then conditioning one hypothesis on each concept. The headline accuracy numbers in Table 2 are computed on held-out test examples, so accuracy is not constructed from the method's own outputs. The diversity numbers in Table 3 use an execution-based equivalence relation (two functions are identical if they return the same outputs for the same inputs), which is independent of the prompting mechanism. The greedy-decoding control in Table 4 directly tests whether the benefit survives without diversity, and the scaling and C/S analyses in Table 5 and Figure 7 use the same held-out evaluation. No parameter is fitted to the reported predictions, no uniqueness theorem is imported from the authors' prior work, and the two self-citations (Lee et al. 2023 and Koh et al. 2024) are peripheral related-work or limitation references, not load-bearing. The statement in Section 3 that LLM-generated lists are 'rarely semantically redundant' is asserted from unquantified empirical investigation, and the paper does not directly measure concept redundancy or hint adherence; this is an unverified mechanism that weakens the causal claim, but it is not circular because the reported diversity and accuracy results are not equal by construction to the method's inputs. The appended Limitations section honestly acknowledges unaddressed human-concept composition and safety concerns, which also do not indicate circularity. Overall, the central claims rest on external test-set evaluation rather than on self-citation or definitional equivalence, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

MoC introduces no new physical or formal entities; the 'concept' is a text hint. The free parameters are all prompt/sampling hyperparameters, and the key extra axiom is the unverified claim that LLM-generated concept lists are semantically non-redundant.

free parameters (4)
  • K (number of hypotheses/concepts) = 8 in main results; varied 4 to 256 in scaling experiments
    Chosen as a compute budget; not fitted, but arbitrary and not swept in main tables.
  • temperature = 1.0
    Selected from the analysis in Section 2.2 showing higher temperature saturates; for temperatures 1.67 and 2.0, top-p p=0.95 is added, changing the sampling distribution.
  • C and S (concepts and samples per concept) = C=8, S=2 at K=16 is reported best
    Table 5 sweeps C and S on the test benchmarks and the best configuration is highlighted; this is test-set hyperparameter selection.
  • top-p p = 0.95
    Used only for temperatures 1.67 and 2.0 in the temperature analysis to avoid degenerate responses.
assumptions (4)
  • domain assumption LLM hypotheses are sampled IID from the prompt distribution
    Justifies the redundancy analysis and the baseline in Section 2.1.
  • ad hoc to paper Autoregressive LLMs tend to generate non-redundant items in a list
    Section 3, Concept Proposal: 'Through empirical investigation, we discovered... rarely semantically redundant.' This is asserted, not measured in the paper.
  • domain assumption Two Python functions are semantically identical if they return the same outputs for the same inputs
    Section 2.1 defines uniqueness this way, but the set of inputs used for the comparison is unspecified.
  • domain assumption The randomly sampled 100/150-instance subsets are representative of each dataset
    Section 4.1 samples subsets without a fixed seed; results may depend on the particular draw.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Diverse Hypotheses for Inductive Reasoning." pith.science (2026). https://pith.science/paper/3C5FHNTV

@misc{pith2026241213422,
  author       = {Pith},
  title        = {Pith review of: Generating Diverse Hypotheses for Inductive Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3C5FHNTV}},
  note         = {Machine review of arXiv:2412.13422}
}
read the original abstract

Inductive reasoning - the process of inferring general rules from a small number of observations - is a fundamental aspect of human intelligence. Recent works suggest that large language models (LLMs) can engage in inductive reasoning by sampling multiple hypotheses about the rules and selecting the one that best explains the observations. However, due to the IID sampling, semantically redundant hypotheses are frequently generated, leading to significant wastage of compute. In this paper, we 1) demonstrate that increasing the temperature to enhance the diversity is limited due to text degeneration issue, and 2) propose a novel method to improve the diversity while maintaining text quality. We first analyze the effect of increasing the temperature parameter, which is regarded as the LLM's diversity control, on IID hypotheses. Our analysis shows that as temperature rises, diversity and accuracy of hypotheses increase up to a certain point, but this trend saturates due to text degeneration. To generate hypotheses that are more semantically diverse and of higher quality, we propose a novel approach inspired by human inductive reasoning, which we call Mixture of Concepts (MoC). When applied to several inductive reasoning benchmarks, MoC demonstrated significant performance improvements compared to standard IID sampling and other approaches.

Figures

Figures reproduced from arXiv: 2412.13422 by the authors.

Figure 1
Figure 1. A motivation for MoC approach. IID sam￾pling frequently generates redundant hypotheses (top). Increasing the temperature leads to frequent occurrences of text degeneration (middle). MoC allows for the gen￾eration of diverse hypotheses without a decline in hy￾pothesis quality (bottom). maintaining their quality. Compared to vanilla IID sampling with the same number of generated hy￾potheses, MoC boosts average accurac… view at source ↗
Figure 3
Figure 3. GPT-4o-mini hypothesis diversity on two do [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. GPT-4o-mini performance on two domains. For the temperature 1.67 and 2.0, we used top-p sam￾pling with p = 0.95. Results are averaged over 5 runs. As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (4 more)
Figure 2
Figure 2. Figure 2: Ratio (%) of degenerate responses. Therefore, we use top-p sampling (Holtzman et al., 2020) with p = 0.95 for these temperature settings, which eliminates degenerate responses entirely. Additionally, by varying the number of sampled responses K, we observe how both K a…
Figure 5
Figure 5. Figure 5: An overview of our Mixture of Concepts approach. We generate [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Example problems in each of four datasets we study. We graphically display the MiniARC examples to [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Test accuracy (%) averaged over 4 datasets. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 21 canonical work pages

  1. [1]

    Samuel Acquaviva. 2024. Overcoming the expressivity-efficiency tradeoff in program induction. Master's thesis, Massachusetts Institute of Technology

    work page 2024

  2. [2]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  3. [3]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. 2024. https://arxiv.org/abs/2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . Preprint, arXiv:2407.21787

    arXiv 2024

  4. [4]

    Jos\' e Cambronero, Sumit Gulwani, Vu Le, Daniel Perelman, Arjun Radhakrishna, Clint Simon, and Ashish Tiwari. 2023. https://doi.org/10.1145/3571226 Flashfill++: Scaling programming by example by cutting to the chase . Proc. ACM Program. Lang., 7(POPL)

    doi:10.1145/3571226 2023

  5. [5]

    Haw-Shiuan Chang, Nanyun Peng, Mohit Bansal, Anil Ramakrishna, and Tagyoung Chung. 2024. Real sampling: Boosting factuality and diversity of open-ended generation via asymptotic entropy. arXiv preprint arXiv:2406.07735

    arXiv 2024

  6. [6]

    Xinyun Chen, Chang Liu, and Dawn Song. 2019. https://openreview.net/forum?id=H1gfOiAqYm Execution-guided neural program synthesis . In International Conference on Learning Representations

    work page 2019

  7. [7]

    Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. 2021. https://proceedings.mlr.press/v139/chen21m.html Spreadsheetcoder: Formula prediction from semi-structured context . In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 16...

    work page 2021

  8. [8]

    Fran c ois Chollet. 2019. On the measure of intelligence. arXiv preprint arXiv:1911.01547

    arXiv 2019

Show all 53 references
  1. [9]

    James Clarke, Dan Goldwasser, Ming-Wei Chang, and Dan Roth. 2010. https://aclanthology.org/W10-2903 Driving semantic parsing from the world ' s response . In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 18--27, Uppsala, Sweden. Ass...

    2010

  2. [10]

    Andrew Cropper. 2019. https://doi.org/10.24963/ijcai.2019/841 Playgol: Learning programs through play . In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 6074--6080. International Joint Conferences on Artificial Int...

    2019 doi

  3. [11]

    Kevin Ellis. 2023. https://openreview.net/forum?id=dVnhdm9MIg Human-like few-shot learning via bayesian reasoning over natural language . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  4. [12]

    Tenenbaum

    Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sabl\' e -Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2021. https://doi.org/10.1145/3453483.3454080 Dreamcoder: bootstrapping inductive program synthesis with wake-sleep library l...

    2021

  5. [13]

    Ryan Greenblatt. 2024. https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt Getting 50\ Technical Report

    2024

  6. [14]

    Sumit Gulwani. 2011. https://doi.org/10.1145/1925844.1926423 Automating string processing in spreadsheets using input-output examples . SIGPLAN Not., 46(1):317–330

    2011

  7. [15]

    Douglas Hofstadter. 1979. Gödel, Escher, Bach: An Eternal Golden Braid. Basic Books, New York

    1979

  8. [16]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In International Conference on Learning Representations

    2020

  9. [17]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representations

    2024

  10. [18]

    Subin Kim, Prin Phunyaphibarn, Donghyun Ahn, and Sundong Kim. 2022. https://openreview.net/forum?id=F4RNpByoqP Playgrounds for abstraction and reasoning . In NeurIPS 2022 Workshop on Neuro Causal and Symbolic AI (nCSI)

    2022

  11. [19]

    Hyukhun Koh, Dohyung Kim, Minwoo Lee, and Kyomin Jung. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.353 Can LLM s recognize toxicity? a structured investigation framework and toxicity metric . In Findings of the Association for Computational Linguistics: EMNLP 2024, p...

    2024 doi

  12. [20]

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. 2024...

    2024 arXiv

  13. [21]

    Lake, Ruslan Salakhutdinov, and Joshua B

    Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. 2015. https://doi.org/10.1126/science.aab3050 Human-level concept learning through probabilistic program induction . Science, 350(6266):1332--1338

    2015 doi

  14. [22]

    Jack Lanchantin, Shubham Toshniwal, Jason Weston, Arthur Szlam, and Sainbayar Sukhbaatar. 2023. https://arxiv.org/abs/2305.00833 Learning to reason and memorize with self-notes . Preprint, arXiv:2305.00833

    2023 arXiv

  15. [23]

    Kang-il Lee, Segwang Kim, and Kyomin Jung. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.425 Weakly supervised semantic parsing with execution-based spurious program filtering . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page...

    2023 doi

  16. [24]

    Wen-Ding Li and Kevin Ellis. 2024. Is programming by example solved by llms? arXiv preprint arXiv:2406.08316

    2024 arXiv

  17. [25]

    Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Yanchi liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, and Wei Cheng. 2024. Scattered forest search: Smarter code space exploration with llms. arXiv preprint arXiv:2411.05010

    2024 arXiv

  18. [26]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. https://openreview.net/forum?id=1qvx610Cu7 Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation . In Thirty-seventh Conference on Neural Informa...

    2023

  19. [27]

    Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. https://proceedings.mlr.press/v28/menon13.html A machine learning framework for programming by example . In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceed...

    2013

  20. [28]

    Suvir Mirchandani, Fei Xia, Pete Florence, brian ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. 2023. https://openreview.net/forum?id=RcZMI8MSyE Large language models as general pattern machines . In 7th Annual Conference on Robot Learning

    2023

  21. [29]

    Melanie Mitchell. 2021. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505(1):79--101

    2021

  22. [30]

    Arseny Moskvichev, Victor Vikram Odouard, and Melanie Mitchell. 2023. The conceptarc benchmark: Evaluating understanding and generalization in the arc domain. arXiv preprint arXiv:2305.07141

    2023 arXiv

  23. [31]

    Augustus Odena, Kensen Shi, David Bieber, Rishabh Singh, Charles Sutton, and Hanjun Dai. 2021. https://openreview.net/forum?id=yHeg4PbFHh \ BUSTLE \ : Bottom-up program synthesis through learning-guided exploration . In International Conference on Learning Representations

    2021

  24. [32]

    Augustus Odena and Charles Sutton. 2020. https://openreview.net/forum?id=rylHspEKPr Learning to represent programs with property signatures . In International Conference on Learning Representations

    2020

  25. [33]

    Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama

    Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. https://openreview.net/forum?id=y0GJXRungR Is self-repair a silver bullet for code generation? In The Twelfth International Conference on Learning Representations

    2024

  26. [34]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? arXiv preprint arXiv:2405.00492

    2024 arXiv

  27. [35]

    Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, and Xiang Ren. 2024. https://openreview.net/forum?id=bNt7oajl2a Phenomenal yet puzzling: Testing inductive reasoning capabilities of langua...

    2024

  28. [36]

    Joshua Stewart Rule. 2020. The child as hacker : building more human-like models of learning. Ph.D. thesis, Massachusetts Institute of Technology

    2020

  29. [37]

    Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, and Dahua Lin. 2025. https://aclanthology.org/2025.coling-main.733/ C ase2 C ode: Scalable synthetic data for code ge...

    2025

  30. [38]

    Kensen Shi, Hanjun Dai, Kevin Ellis, and Charles Sutton. 2022. https://openreview.net/forum?id=qhC8mr2LEKq Crossbeam: Learning to search in bottom-up program synthesis . In International Conference on Learning Representations

    2022

  31. [39]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314

    2024 arXiv

  32. [40]

    Elizabeth S Spelke and Katherine D Kinzler. 2007. Core knowledge. Developmental science, 10(1):89--96

    2007

  33. [41]

    Wangtao Sun, Haotian Xu, Xuanqing Yu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. 2024. https://doi.org/10.18653/v1/2024.acl-long.150 I t D : Large language models can teach themselves induction through deduction . In Proceedings of the 62nd Annual Meeting of the Association ...

    2024 doi

  34. [42]

    Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. 2024. Code repair with llms gives an exploration-exploitation tradeoff. arXiv preprint arXiv:2405.17503

    2024 arXiv

  35. [43]

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024 a . Planning in natural language improves llm search for code generation. arXiv preprint arXiv:2409.03733

    2024 arXiv

  36. [44]

    Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah Goodman. 2024 b . https://openreview.net/forum?id=G7UtIGQmjm Hypothesis search: Inductive reasoning with language models . In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Taylor Webb, Keith J Holyoak, and Hongjing Lu. 2023. Emergent analogical reasoning in large language models. Nature Human Behaviour, 7(9):1526--1541

    2023

  38. [46]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  39. [47]

    Yeming Wen and Swarat Chaudhuri. 2024. https://openreview.net/forum?id=sp8wHIsnu9 Synthesize, partition, then adapt: Eliciting diverse samples from foundation models . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  40. [48]

    Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. 2024. https://aclanthology.org/2024.eacl-long.13/ Language models as inductive reasoners . In Proceedings of the 18th Conference of the European Chapter of the Association for C...

    2024

  41. [49]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  42. [50]

    Chi, Quoc V Le, and Denny Zhou

    Huaixiu Steven Zheng, Swaroop Mishra, Xinyun Chen, Heng-Tze Cheng, Ed H. Chi, Quoc V Le, and Denny Zhou. 2024. https://openreview.net/forum?id=3bq3jsvcQ1 Take a step back: Evoking reasoning via abstraction in large language models . In The Twelfth International Conference on L...

    2024

  43. [51]

    Yue Zhou, Yada Zhu, Diego Antognini, Yoon Kim, and Yang Zhang. 2024. https://doi.org/10.18653/v1/2024.naacl-long.153 Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models . In Proceedings of the 2024 Confer...

    2024 doi

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.