Pith. sign in

REVIEW 3 major objections 6 minor 37 references

Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Randomly sampled grammar tasks show LLMs fall short of n-gram baselines.

desk verdict Solid core measurement, overreaching abstract: LLMs underperform n-grams on 3-state DFA induction under the tested prompts, but 'uniformly' and 'wholly unseen' overstate the evidence. read the letter →

arxiv 2501.02825 v6 pith:IZHQ6DTS submitted 2025-01-06 cs.LG

Kavi Gupta , Kate Sanders , Armando Solar-Lezama This is my paper · ORCID
classification cs.LG
keywords in-contextlearninglargelanguagemodelsdeterministicfiniteautomataregularlanguagesn-grambaselineschain-of-thoughtpromptingreasoningnovelty
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to measure in-context learning (ICL) in isolation from everything an LLM has memorized. The authors generate tasks from randomly sampled 3-state deterministic finite automata, so every language is simple but unfamiliar, and compare LLMs to n-gram baselines that have no world model and no pretraining. Across a broad set of open-weight and proprietary models, with the better of two structure-free prompts chosen per model, LLMs uniformly underperform the baselines: below a 4-gram on the transducer task and, at best, just under the 2-gram on sequence completion. The result matters because it suggests that on entirely novel language structure, foundation-model in-context learning is not competitive with elementary token statistics, even with chain-of-thought prompting.

What carries the argument

The machinery is a benchmark built from randomly sampled 3-state deterministic finite automata over a three-letter alphabet. A DFA is the simplest finite-state machine that accepts or rejects strings, and three states is the smallest case with hidden state beyond the accept/reject output. Random sampling from the space of such DFAs is what removes training-corpus bias: the authors deliberately avoid canonical grammars. The argument is carried by comparing LLM accuracy under the best of two structure-free prompts against unparameterized n-gram baselines, which match token clusters rather than build world models. The 3-state design also lets the authors include a brute-force upper bound over all DFAs, showing the tasks are solvable in principle.

What would settle it

The claim would collapse if a single LLM, given only example strings or transducer traces with no mention of grammar, DFA, rooms, or portals, matched or beat the 6-GRAM transducer baseline of 93.5% (or the 2-GRAM sequence-completion baseline of 83.3%) on the same sampled DFAs.

Watch

Extended reading notes

Core claim

The central claim is that LLMs uniformly underperform n-gram models at inducing simple but entirely unfamiliar languages from examples. The authors test next-token prediction on two tasks generated from randomly sampled 3-state DFAs with a three-letter alphabet: completing a prefix so the full string is accepted, and predicting the next transducer output bit. Taking the maximum over the two main prompt formats, BASIC and BASIC-COT, no LLM reaches a 4-gram on the transducer task, and on sequence completion the best LLM stays just below the 2-gram baseline, with all comparisons against 4- to 6-grams statistically significant. Because the n-gram baselines have no hidden state and no world model, the paper argues the failure cannot be blamed on task difficulty or on missing world knowledge; it is a limit of in-context learning on novel language structure.

Load-bearing premise

The load-bearing premise is that the finite set of prompts tried, mainly BASIC and BASIC-COT with the best result taken, adequately elicits each model's in-context learning ability, so that low scores reflect ICL limitations rather than prompt-format effects.

Editorial extensions

If this is right

  • If the central claim is correct, in-context learning on genuinely novel tasks is weaker than primitive n-gram statistics for inducing simple regular languages.
  • Strong performance on canonical language and reasoning tasks should not be read as evidence of a general theory of language; the authors conclude that LLMs have learned models of particular languages rather than language in general.
  • Chain-of-thought prompting does not overcome the novelty gap when the problem structure is withheld, although explicit DFA framing does let reasoning models approach the brute-force upper bound.
  • The benchmark construction generalizes: sampling languages from larger DFA state counts guarantees by a pigeonhole argument that the sampled languages were not seen in training, providing a scalable way to evaluate ICL.
  • If current models are the target, the paper's results imply that ICL is not yet a path to replacing all machine learning with in-context learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the scaling experiment in Appendix D suggests LLM ICL improves with more examples on the transducer task but appears to converge near n-gram-level performance rather than toward the DFA-solving brute-force bound; a direct test would be to plot accuracy against example count past 600 and compare the asymptote to 6- and 7-gram baselines.
  • My inference: the RED-GREEN framing results show that how a task is described can move performance by roughly 20 points, so the same benchmark could be re-run with an automated search over paraphrased structure-free prompts before accepting a model-specific ICL ceiling.
  • My inference: because n-gram baselines are cheap and parameter-free, this evaluation design could be adopted as a routine sanity check in ICL benchmarks to distinguish genuine induction from recall of familiar patterns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces two benchmarks for evaluating in-context learning (ICL) on randomly sampled 3-state DFA languages: a sequence completion task and a transducer next-token prediction task. With 30 in-context examples per instance, it compares open-weight and proprietary LLMs under BASIC and BASIC-COT prompts against parameter-free baselines (random, common-suffix, n-gram, brute-force). The main findings are that on sequence completion all tested LLMs fall below the 2-GRAM S baseline, on transducer all tested LLMs fall below the 4-GRAM T baseline under the tested ICL-only prompts, and comparisons against 4- to 6-gram baselines are statistically significant. The paper also reports that chain-of-thought rarely helps on the transducer task, that control prompts revealing DFA structure let reasoning models such as gpt-5 and o3-mini reach near-brute-force transducer accuracy, and it includes a detailed case study on the sum-modulo-3 DFA. The authors conclude that LLM ICL is not competitive with primitive n-gram statistics on wholly unfamiliar language reasoning tasks.

Significance. If the headline result held as stated, it would be a clean, well-controlled negative result about the limits of ICL on novel regular languages, with implications for how the community interprets few-shot performance on unfamiliar tasks. The benchmark design has real virtues: random DFA sampling reduces canonical-task bias; the n-gram, random, common-suffix, and brute-force baselines are parameter-free and defined independently of the models under test; the RANDOM S baseline explicitly measures rejection-sampling bias; and the paper includes supplementary controls for temperature, tokenization, non-answers, and example count. Statistical reporting with 95% confidence intervals and paired bootstrap significance tests is careful, and the sum-mod-3 case study is a useful qualitative bridge between LLM errors and n-gram behavior. My main reservation is not with the measurements but with the scope of the conclusion: the abstract and Section 5.2 state a general limitation of ICL, whereas the experiments establish underperformance for two ICL-only prompt formats, and for most models only one format was actually run. This is an elicitation gap, not a circularity or baseline problem.

major comments (3)
  1. [§4, Table 2 caption, Abstract] The abstract claims underperformance 'both when used as next token predictors and in chain-of-thought,' but Table 2's caption states 'we only used BASIC on other models' and Section 4 defines the main results as the maximum over BASIC and BASIC-COT. Chain-of-thought was therefore not evaluated for the large majority of models, including all open-weight and code models and the gpt-3.5 models. The chain-of-thought claim should either be removed from the abstract or backed by running BASIC-COT on the remaining models; as written, the abstract reports evidence that does not exist.
  2. [§4, Table 2, §5.2, §6] The central conclusion that 'LLM ICL is unable to perform well at language inference' rests on the assumption that BASIC and BASIC-COT adequately elicit each model's in-context learning ability. The paper itself concedes in Section 4 that 'no finite set of prompts will be fully sufficient to capture all possible model behavior,' and Table 2 shows large prompt-conditioned swings for the same model (o3-mini on Transducer: 72.8 on BASIC T, 86.1 on DFA-COT T, 92.4 on RED-GREEN T; gpt-5: 83.6 on BASIC T versus 96.7 on DFA-COT T). The control prompts disclose the DFA structure and therefore cannot serve as ICL-only evidence, but they demonstrate that instruction format alone can move performance by more than the gap to 4-GRAM T. Since the headline claim is about a general limitation of ICL, the paper needs either a systematic search over ICL-only prompt variations (e.g., separators, answer format, example ordering, instruction phrasing, or diverse reasoning prompts that do not disclose structure) or a conclusion explicitly restricted to the tested prompts. The Section 6 caveat about the finite prompt set is in the right direction, but the abstract does not carry it.
  3. [§5.2, Table 1, Table 2] The sentence 'we find that in general all LLMs underperform a 4-GRAM T model' should be qualified as holding for the ICL-only prompt family. In Table 2, gpt-5 reaches 96.7 on DFA-COT T and 96.6 on RED-GREEN T, and o3-mini reaches 92.4 on RED-GREEN T, both above the 4-GRAM T accuracy of 91.1. These results do not refute the ICL claim because those prompts reveal the underlying DFA structure, but the unqualified sentence is easy to misread as a statement about raw model capability that the paper's own data contradict.
minor comments (6)
  1. [Introduction, first paragraph] The sentence 'LLMs do not posses the ability to generalize' contains a typo: 'posses' should be 'possess'.
  2. [§2.1, Related Work] The sentence beginning 'Some have suggested that indicate that LLMs may possess emergent language reasoning abilities' is ungrammatical due to the duplicated 'that indicate that' clause and should be rephrased.
  3. [Table 1 caption] The caption should state explicitly that the table reports the maximum over BASIC and BASIC-COT for models where both were run, and BASIC only otherwise; this aggregation rule currently appears only in the body text.
  4. [Appendix J] The sentence 'some models requireed the use of 4 in parallel' contains a typo: 'requireed' should be 'required'.
  5. [NeurIPS Checklist, Sections 5 and 13] Section 13 says no new assets are released, while Section 5 says a code zip file is attached; these two statements should be reconciled so that the reproducibility status is unambiguous.
  6. [Abstract] The abstract's phrase 'underperform n-gram models' is ambiguous because on the Transducer task many LLMs outperform 2-GRAM T; the body's precise claim concerns 4-GRAM T and above, and the abstract should match that specificity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found: all headline comparisons are direct measurements against external, parameter-free n-gram baselines, and cited support comes from independent prior work.

full rationale

The paper's claimed chain is: define randomly sampled DFA tasks, evaluate LLMs with fixed prompt families, compute accuracy, and compare against n-GRAM, RANDOM, NULL, COMMON-SUFFIX, and BRUTE-FORCE baselines. None of these baselines is fitted to the LLM results, and none of the reported accuracies is derived from a fitted parameter that is then renamed a prediction; the n-gram baselines are deterministic algorithms over the same in-context examples and the target prefix. The pilot-sampling bias is explicitly controlled by the RANDOM S baseline rather than by post-hoc adjustment, so the benchmark's difficulty measure is not constructed from the target outcome. The one self-citation in the paper (Weir et al., 2023/2024 in Section 2.2) is a related-work pointer and is not used to justify any result. The external citations used as premises (Akyürek et al., 2024; Svete & Cotterell, 2024) come from different research groups and support representability or learnability claims; they do not define the measured quantities. The paper explicitly acknowledges the prompt-elicitation limitation in Section 4: 'no finite set of prompts will be fully sufficient to capture all possible model behavior,' which is a scope caveat about eliciting model capability, not a circular definition. Gaps such as Table 2 showing only BASIC for most open-weight models while the abstract mentions chain-of-thought are evidential gaps affecting generality, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No constants are fitted to produce the headline numbers; the comparison is a direct measurement against external baselines. The listed parameters are hand-chosen design values, each stress-tested in appendices. The axioms are the distributional representativeness of the DFA sample, the representativeness of the finite prompt set, and two cited transformer-capability results used to argue the tasks are solvable in principle. No new entities are introduced.

free parameters (3)
  • n-gram order n = 2 to 6 (scanned)
    Baseline hyperparameter swept from 2 to 6; the headline conclusion holds for n >= 4 on both tasks, so no single value is fitted to support the claim.
  • example count per task instance = 30
    Hand-chosen; Appendix D reruns with 300 and 600 examples and the qualitative conclusion (the LLM at no point beats the best n-gram) persists, so the value is not load-bearing.
  • completion length bound = 1 to 5 characters
    Fixed task design for sequence completion; the n-gram and LLM evaluations share the same bound, so this is a definitional choice rather than a fitted constant.
assumptions (4)
  • domain assumption The rejection-sampled 3-state DFA distribution represents simple languages without bias toward canonical languages.
    Section 3.2 and Appendix A.1; footnote 1 admits the pigeonhole guarantee does not cover 78,786 3-state DFAs, so the 'wholly unseen' premise is weaker than stated.
  • domain assumption Maximum accuracy over the finite prompt set (BASIC, BASIC-COT, plus structure-revealing controls) fairly represents each model's ICL ability.
    Section 4 states 'no finite set of prompts will be fully sufficient'; Table 2 shows 10 to 30 point swings across prompts for the same model, e.g., o3-mini on Transducer ranges from 72.8 to 92.4, so this premise is load-bearing.
  • standard math Transformers can represent n-gram models, so a transformer architecture could in principle match n-gram performance.
    Invoked in Section 3.3 and cited to Svete and Cotterell (2024); used to argue that underperformance is not an architectural ceiling.
  • domain assumption Transformers trained on DFA traces can learn to in-context-learn regular languages, so the 3-state tasks are solvable by transformers in principle.
    Section 2.4 cites Akyurek et al. (2024) for 4 to 12 state DFAs; the paper extrapolates this to foundation LLMs on 3-state DFAs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning." pith.science (2026). https://pith.science/paper/IZHQ6DTS

@misc{pith2026250102825,
  author       = {Pith},
  title        = {Pith review of: Randomly Sampled Language Reasoning Problems Elucidate Limitations of In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZHQ6DTS}},
  note         = {Machine review of arXiv:2501.02825}
}
read the original abstract

While LLMs have revolutionized the field of machine learning due to their high performance on a strikingly wide range of problems, they are also known to hallucinate false answers and underperform on less canonical versions of the same tasks. There are several emerging theories of LLM performance, among them that LLMs lack world modeling ability, that they have an undesirable bias towards an autoregressive prior, and that they struggle on more novel problems. The existing literature on LLM input novelty has focused on tasks of relatively high complexity, studying perturbations of canonical but complex problems. In this paper, we attempt to minimize complexity in order to isolate novelty as a factor in LLM underperformance and investigate the power of in-context-learning. To this end, we consider an extremely simple domain: next token prediction on simple language tasks. The twist is that these language tasks are wholly unseen, as they are randomly drawn from a large, parsimoniously defined set of languages arising from simple grammar rules. This experimental setup allows us to evaluate ICL independently of models' parametric knowledge. We find that LLMs uniformly underperform n-gram models on this task, both when used as next token predictors and in chain-of-thought.

Figures

Figures reproduced from arXiv: 2501.02825 by the authors.

Figure 1
Figure 1. We sample randomly generated lan￾guages to test LLMs by sampling deterministic finite automata (DFAs). (a) The DFA shown here, modeling the sum modulo 3 operation (with abc representing 0, 1, and 2 respectively), can be used to accept or reject strings from a 3-character al￾phabet. Accepted strings belong to the grammar; rejected strings do not. We evaluate models on their ability to (b) act as a transducer, recogni… view at source ↗
Figure 2
Figure 2. An illustration of Chomsky’s hierarchy of lan [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Transducer and sequence completion results plotted against each other. Points are the [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Transducer results by difficulty class. We classify each DFA based on which of the baselines [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Results on Sum Modulo 3 DFA. (a) MB=mistral-nemo-minitron-8B/BASICT , CR=claude-3.5/RED-GREENT . Venn diagram of errors (out of 1000). Labeled percentages are ac￾curacies. (b) Results of qualitative analysis, out of 30 in both cases. We investigate the transducer task …
Figure 6
Figure 6. Figure 6: Results on Sum Modulo 3 DFA under trivial / nontrivial conditions. Percentages are [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Accuracy by number of examples, as we vary the number from 30 to 600 (note the log [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Significance of comparisons between rows of Table 1. Results in blue and orange are [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 7 canonical work pages

  1. [3]

    Mathqa: To- wards interpretable math word problem solving with operation-based formalisms.arXiv preprint arXiv:1905.13319,

    Amini, A., Gabriel, S., Lin, P., Koncel-Kedziorski, R., Choi, Y ., and Hajishirzi, H. Mathqa: To- wards interpretable math word problem solving with operation-based formalisms.arXiv preprint arXiv:1905.13319,

    arXiv 1905

  2. [4]

    Understanding in-context learning in transformers and llms by learning to learn discrete functions.arXiv preprint arXiv:2310.03016,

    10 Bhattamishra, S., Patel, A., Blunsom, P., and Kanade, V . Understanding in-context learning in transformers and llms by learning to learn discrete functions.arXiv preprint arXiv:2310.03016,

    arXiv

  3. [5]

    Leveraging code to improve in-context learning for semantic parsing.arXiv preprint arXiv:2311.09519,

    Bogin, B., Gupta, S., Clark, P., and Sabharwal, A. Leveraging code to improve in-context learning for semantic parsing.arXiv preprint arXiv:2311.09519,

    arXiv

  4. [6]

    What languages are easy to language-model? a perspective from learning probabilistic regular languages.arXiv preprint arXiv:2406.04289,

    Borenstein, N., Svete, A., Chan, R., Valvoda, J., Nowak, F., Augenstein, I., Chodroff, E., and Cotterell, R. What languages are easy to language-model? a perspective from learning probabilistic regular languages.arXiv preprint arXiv:2406.04289,

    arXiv

  5. [9]

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al

    URL https://huggingface.co/deepseek-ai/ deepseek-coder-33b-instruct. Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. A survey on in-context learning.arXiv preprint arXiv:2301.00234,

    arXiv

  6. [10]

    Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes.arXiv preprint arXiv:2312.14890,

    Fan, L., Hua, W., Li, L., Ling, H., and Zhang, Y . Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes.arXiv preprint arXiv:2312.14890,

    arXiv

  7. [11]

    Hazra, R., Venturato, G., Martires, P

    URLhttps://huggingface.co/google/gemma-7b. Hazra, R., Venturato, G., Martires, P. Z. D., and De Raedt, L. Can large language models reason? a characterization via 3-sat.arXiv preprint arXiv:2408.07215,

    arXiv

  8. [12]

    Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874,

    arXiv

Show all 37 references
  1. [14]

    and Chang, K

    Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: A survey.arXiv preprint arXiv:2212.10403,

    arXiv

  2. [15]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186,

    arXiv

  3. [16]

    J., and Fard, F

    Joel, S., Wu, J. J., and Fard, F. H. A survey on llm-based code generation for low-resource and domain-specific programming languages.arXiv preprint arXiv:2410.03981,

  4. [17]

    Decomposed prompting: A modular approach for solving complex tasks.arXiv preprint arXiv:2210.02406,

    Khot, T., Trivedi, H., Finlayson, M., Fu, Y ., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks.arXiv preprint arXiv:2210.02406,

    arXiv

  5. [20]

    What makes good in-context examples for gpt-3?arXiv preprint arXiv:2101.06804,

    Liu, J., Shen, D., Zhang, Y ., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3?arXiv preprint arXiv:2101.06804,

    arXiv

  6. [21]

    T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T

    McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained to solve.arXiv preprint arXiv:2309.13638,

    arXiv

  7. [23]

    URL https://huggingface.co/nvidia/ Mistral-NeMo-Minitron-8B-Base. OpenAI. Gpt 3.5 turbo, 2024a. URL https://openai.com/index/ new-embedding-models-and-api-updates/. OpenAI. Gpt-4o system card, 2024b. URLhttps://arxiv.org/abs/2410.21276. OpenAI. Openai gpt-5 system card, 2025a....

    arXiv

  8. [25]

    L., Gardner, M., and Singh, S

    Razeghi, Y ., Logan IV , R. L., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot reasoning.arXiv preprint arXiv:2202.07206,

    arXiv

  9. [26]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615,

    arXiv

  10. [27]

    and Cotterell, R

    Svete, A. and Cotterell, R. Transformers can represent n-gram language models.arXiv preprint arXiv:2404.14994,

    arXiv

  11. [28]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    Team, Q. Qwen2 technical report.arXiv preprint arXiv:2407.10671,

    arXiv

  12. [30]

    Benchmarking compositionality with formal languages.arXiv preprint arXiv:2208.08195,

    Valvoda, J., Saphra, N., Rawski, J., Williams, A., and Cotterell, R. Benchmarking compositionality with formal languages.arXiv preprint arXiv:2208.08195,

    arXiv

  13. [31]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022a

    Wei, J., Tay, Y ., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022a. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, ...

    arXiv

  14. [32]

    D., Tafjord, O., Jansen, P., Clark, P., et al

    Weir, N., Sanders, K., Weller, O., Sharma, S., Jiang, D., Zhang, Z., Mishra, B. D., Tafjord, O., Jansen, P., Clark, P., et al. Enhancing systematic decompositional natural language inference using informal logic.arXiv preprint arXiv:2402.14798,

    arXiv

  15. [33]

    Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks

    Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y . Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. InProceedings of the 2024 Conference of the North American Chapter ...

    2024

  16. [34]

    F., Alon, U., Neubig, G., and Hellendoorn, V

    Xu, F. F., Alon, U., Neubig, G., and Hellendoorn, V . J. A systematic evaluation of large language models of code. InProceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pp. 1–10, 2022a. Xu, Z., Niethammer, M., and Raffel, C. A. Compositional gene...

    2023 arXiv

  17. [35]

    and Koller, A

    13 Yao, Y . and Koller, A. Structural generalization is hard for sequence-to-sequence models.arXiv preprint arXiv:2210.13050,

    arXiv

  18. [36]

    Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625,

    Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625,

    arXiv

  19. [37]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  20. [1959]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    arXiv

  21. [2005]

    Language models as models of language.arXiv preprint arXiv:2408.07144,

    Millière, R. Language models as models of language.arXiv preprint arXiv:2408.07144,

    arXiv

  22. [2015]

    Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191,

    Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems?arXiv preprint arXiv:2103.07191,

    arXiv

  23. [2018]

    Y ., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y ., and Ren, X

    Lin, B. Y ., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y ., and Ren, X. Commongen: A constrained text generation challenge for generative commonsense reasoning.arXiv preprint arXiv:1911.03705,

    1911 arXiv

  24. [2019]

    Few-shot adaptation for parsing contextual utterances with llms.arXiv preprint arXiv:2309.10168,

    Lin, K., Xia, P., and Fang, H. Few-shot adaptation for parsing contextual utterances with llms.arXiv preprint arXiv:2309.10168,

    arXiv

  25. [2021]

    and Zhang, Y

    Hua, W. and Zhang, Y . System 1+ system 2= better world: Neural-symbolic chain of logic reasoning. InFindings of the Association for Computational Linguistics: EMNLP 2022, pp. 601–612,

    2022

  26. [2022]

    In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973,

    Akyürek, E., Wang, B., Kim, Y ., and Andreas, J. In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973,

    arXiv

  27. [2023]

    URL https://github.com/meta-llama/llama3/blob/ main/MODEL_CARD.md. AI@Meta. Llama 3.1 70b, 2024a. URL https://huggingface.co/meta-llama/Llama-3. 1-70B. AI@Meta. Llama 3.1 8b, 2024b. URL https://huggingface.co/meta-llama/Llama-3.1-8B . AI@Meta. Llama 3.1 8b instruct, 2024c. URL...

    arXiv

  28. [2024]

    Training neural networks as recognizers of formal languages.arXiv preprint arXiv:2411.07107,

    Butoi, A., Khalighinejad, G., Svete, A., Valvoda, J., Cotterell, R., and DuSell, B. Training neural networks as recognizers of formal languages.arXiv preprint arXiv:2411.07107,

    arXiv

  29. [2025]

    Large language models still can’t plan (a benchmark for llms on planning and reasoning about change)

    Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S. Large language models still can’t plan (a benchmark for llms on planning and reasoning about change). InNeurIPS 2022 Foundation Models for Decision Making Workshop,

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.