Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring

Alex Heyman; Joel Zylberberg

REVIEW 2 major objections 6 minor 23 references

Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring

T0 review · 2 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that graph coloring separates step-by-step reasoning from possibility-space exploration, and that all four standard LLMs exceed 60% error on hard types while even reasoning models miss trivial 4-vertex cases.

desk verdict A clean, reproducible LLM benchmark on graph coloring whose headline error-rate claims hold up, though the greedy-score difficulty axis is less firmly grounded than the abstract suggests. read the letter →

arxiv 2502.07087 v1 pith:XDDEJFNS submitted 2025-02-10 cs.LG

Alex Heyman , Joel Zylberberg This is my paper · ORCID

classification cs.LG

keywords graphcoloringLLMreasoningpossibilityspaceexplorationgreedyscoresemanticframinglargemodelscompositional

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper uses small graph-coloring puzzles—color the vertices of a 4-to-8-vertex graph with 2 to 4 colors so adjacent vertices differ—as a controlled test bed for LLM reasoning. It tries to establish two things: that solving these puzzles requires systematic step-by-step reasoning plus exploring alternative color assignments when a greedy choice fails, and that current models fail at the second requirement in a measurable way. Across four semantic framings of the same problems, the four standard LLMs exceed 60% error on the difficult problem types in every frame, while the two reasoning-specialized models stay below 20% error but still exceed 10% on the hardest colorable cases. No tested model reaches 0% error even on trivial 4-vertex 2-coloring problems, so the paper argues there is no complexity range in which these models can be trusted to reason reliably.

What carries the argument

The central object is the greedy score $g$, the success rate of a deliberately imperfect randomized greedy coloring algorithm run 10,000 times per problem. Colorable problems are binned into $g \ge 0.9$, $0.5 \le g < 0.9$, and $g < 0.5$; uncolorable problems are split into complete-uncolorable and diffuse-uncolorable categories by presence of a $K_{k+1}$ subgraph. The greedy score does the work of ordering problems by how much backtracking and exploration a solver needs, and the paper's difficulty trends are defined against those thresholds.

What would settle it

A single class of colorable problems where the greedy score is low (below $g < 0.5$) but LLM error is at chance, or a class with high greedy score (above $g \ge 0.9$) where LLM error exceeds 60%, would break the proposed difficulty axis. More directly: re-run the 4v2c evaluation with far more than five repetitions per graph; if any model reaches 0% error across all 63 non-edgeless 4-vertex graphs, the paper's 'no model reaches perfect accuracy' claim fails.

Watch

Extended reading notes

Core claim

The central claim is that graph coloring separates linear step-by-step reasoning from possibility-space exploration, and that current LLMs are much weaker at the latter. Problems are procedurally generated across five sets (4v2c, 5v2c, 6v3c, 7v3c, 8v4c), with colorable problems assigned a greedy score $g$: the fraction of 10,000 runs of a randomized greedy algorithm that find a valid coloring. Error rates rise as $g$ falls, and the jump from $0.5 \le g < 0.9$ to $g < 0.5$ is proportionally larger for o1-mini and DeepSeek-R1 than for standard LLMs; the paper reads this as evidence that reinforcement-trained reasoning models have not improved at exploring the space of possibilities as much as they have improved at straight-line derivation. All four standard LLMs exceed 60% error on difficult 8v4c types in all frames, and no model reaches perfect accuracy on 4-vertex 2-coloring problems.

Load-bearing premise

The entire difficulty argument rests on the greedy score being a valid and transferable measure of how hard a graph-coloring problem is for a large language model, with the three chosen thresholds meaningfully separating easy from hard reasoning.

Editorial extensions

If this is right

If the difficulty trends are right, no current model has a trustworthy operating range: even the simplest problem set, 4v2c, produces nonzero error, so autonomous systems that perform compositional reasoning steps cannot assume a correct answer.
The larger proportional error jump at low greedy scores for o1-mini and DeepSeek-R1 implies that adding more chain-of-thought tokens will not by itself fix the missing possibility-space exploration; the paper explicitly advises against relying on longer reasoning traces.
Because graph coloring instances are cheap to generate at any complexity, this benchmark can be scaled to test where LRM performance collapses toward chance, a question the paper leaves open.
Framing effects that differ across models mean single-frame accuracy numbers are unstable; evaluation that repeats problems across semantic frames gives a more complete picture of a model's reasoning robustness.
The Math (demanding) frame, which tells the model to produce a coloring or say impossible, biases models away from answering 'impossible,' so small prompt perturbations can change measured reasoning ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: If the greedy score tracks LLM difficulty, the same measure should predict error on other constraint-satisfaction tasks that require backtracking, such as Sudoku or scheduling; a cheap test would be to regress LLM error against greedy scores computed for those domains.
Inference: The persistence of nonzero error on 4v2c suggests these models are not performing exhaustive constraint propagation; one concrete probe for this is comparing model errors against search-tree size of a complete solver, predicting larger errors where the search tree is deeper rather than where the graph is denser.
Inference: The paper's framing effects might reflect training-data prevalence; a direct extension would compare the same graph-coloring instance expressed in several equally common real-world frames to see whether error tracks frame familiarity.
Inference: The observed error gap between standard and reasoning models may narrow or widen as inference-time compute grows; a testable extension is measuring o1-mini and DeepSeek-R1 accuracy on 8v4c g<0.5 instances as a function of response length, to see whether extra tokens are spent on genuine search.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look. This is a clean, honest evaluation paper, and the headline numbers are basically right. The authors test six models on procedurally generated graph coloring problems in four semantic frames, with a public repo including problems, prompts, and responses. The main findings—standard LLMs exceed 60% error on the hard 8v4c categories in every frame, o1-mini and DeepSeek-R1 stay under 20%, and no model hits 0% even on 4-vertex 2-coloring—are well supported by the figures and appendix tables. The exhaustive enumeration for 4v2c and 5v2c, the repeated trials, the manual review of unparseable answers, and the Clopper-Pearson intervals all point to careful experimental hygiene. The framing effects are real and vary sensibly across models, which is a useful addition to the contamination and robustness literature.

The soft spot is the difficulty measure. The greedy score g is an independent, non-circular proxy, which is good, but the paper never validates that it tracks what makes these problems hard for LLMs. The authors themselves flag the 'puzzling' pattern in Section 3.1: error jumps from g>=0.9 to 0.5<=g<0.9 but barely moves from there to g<0.5. That suggests g is riding on something else, like edge density or chromatic number. The LRM comparison would be cleaner if g categories were validated or if they reported the same error-rate-by-edge-count analyses as the primary view. Appendix G does a decent job of equalizing the problem sets by comparing on identical inputs, but the central 'possibility-space exploration' interpretation still leans on g.

Two smaller points: the LRMs are tested on only two frames and a subset of edge counts for cost reasons, which the authors state but should make more prominent; and the paper would be stronger with significance tests for the framing and difficulty trends instead of only confidence intervals. The closing advice about test-time compute is reasonable but goes beyond the evidence; it's clearly labeled as opinion, so it's a minor overreach.

Who should read it: anyone working on LLM reasoning evaluation, especially people building benchmarks with procedural generation. It's not a theory paper; it's a careful measurement. It deserves a serious referee. I'd send it out and ask the authors to (1) spend a page validating g—e.g., correlation with edge density, chromatic number, or alternative greedy variants—and (2) move exact numbers with intervals into the text.

Referee Report

2 major / 6 minor

Summary. The paper evaluates six LLMs (four standard LLMs — Llama 3.1 405B, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro — and two reasoning models, o1-mini and DeepSeek-R1) on small graph k-coloring problems (4v2c through 8v4c), presented in up to four semantic frames. Problems are categorized by a randomized greedy algorithm's success rate (greedy score g) into three difficulty tiers for colorable graphs, and by complete vs. diffuse uncolorability for uncolorable graphs. The main empirical findings are that standard LLMs exceed 60% error on the difficult colorable tiers in all frames, LRMs stay below 20% error but exceed 10% on the hardest tier, all models show nonzero error even on 4-vertex 2-coloring, and framing effects are substantial and model-dependent. The paper interprets the results as evidence of limited systematic step-by-step reasoning and possibility-space exploration in LLMs, with LRMs improving on the former but not fully on the latter.

Significance. The strengths are concrete: the small problem sets are exhaustively enumerated or randomly sampled with stated criteria, responses were collected with a public repository, repeated trials with model-specific temperatures, Clopper-Pearson confidence intervals in the appendices, and a manual review protocol for unparsable responses. The multi-frame design is a useful contribution for studying semantic biases, and graph coloring is a sensible, contamination-resistant procedural reasoning benchmark. If the difficulty-proxy issue in Major Comment 1 is resolved, the comparative results would be a valuable addition to the LLM-reasoning literature. As it stands, the headline error rates are competently measured, but their interpretation as evidence about 'difficulty' and 'possibility-space exploration' is not yet supported.

major comments (2)

[Section 2.2 and Section 3.1] The difficulty categorization built on the greedy score g (Section 2.2) is the backbone of the headline claims that model error rates exceed 60% on 'difficult problem types' and of the discussion's conclusion that LRMs underperform standard LLMs at possibility-space exploration. The paper never validates that g transfers to LLM difficulty; it is the success rate of a particular randomized greedy algorithm, and the thresholds 0.9 and 0.5 are arbitrary. Section 3.1 itself reports a 'puzzling' pattern under a greedy-accurate model of LLM behavior: the error-rate jump from g >= 0.9 to 0.5 <= g < 0.9 is much larger than the jump from 0.5 <= g < 0.9 to g < 0.5, and the authors attribute part of the increase to 'some other form of difficulty increase correlated with g substantially below 1.' That admission shows that g is not the operative difficulty axis for these models. Because the abstract's error-rate claims are expressed in terms of g-categories, the current text leaves open the possibility that the observed separations are driven by confounded covariates such as edge density, chromatic number, or number of valid colorings. I request either (a) a validation of g against LLM error rates controlling for n, k, and edge count (or another structural difficulty measure), or (b) a systematic softening of the 'difficulty' and 'possibility-space exploration' interpretations so that the paper reports error rates as a function of edge count and other graph features without committing to g as the difficulty axis.
[Section 2.4 and Abstract] The abstract's comparative statement — standard LLMs >60% error on difficult types, o1-mini >15%, R1 >10% — pools results from two different problem distributions: standard LLMs were tested on the full problem sets, while the LRMs were tested on edge-selected subsets (Section 2.4). Although Appendix G displays standard-LLM error rates on the LRM-selected subsets and thus partially mitigates the confound, the headline numbers in the main text and abstract are not drawn from that matched comparison. The main text should either present the matched Appendix G numbers as the basis for the headline comparison or explicitly state that the abstract percentages come from different problem sets; as written, a reader could mistakenly infer a like-for-like comparison.

minor comments (6)

[Section 2.5 and Appendix C] At temperature 0, repeated trials are not independent draws; the paper should report the rate of identical responses or use per-problem accuracy. This does not change the qualitative conclusions but affects the stated confidence intervals.
[Abstract and Section 2.4] The abstract's parenthetical (>15% for o1-mini and >10% for R1) does not state that these figures are over two frames, while the >60% claim for standard LLMs is over four frames; clarify to avoid misreading.
[Section 3.1] Report the counts of Llama refusals; the statement that they made 'little practical difference' is not quantified.
[Section 1.1] The 'superexponentially' claim about the number of graphs would be clearer as 2^{n(n-1)/2} for labeled graphs, and 'superexponential' is ambiguous without specifying the base.
[Section 2.2 and Appendix A] The greedy score is a Monte Carlo estimate over 10,000 trials; a sentence on its standard error or the stability of categorization near thresholds would strengthen the method.
[Section 4] The phrase 'ala Dziri et al. (2024)' in the fourth paragraph should be 'à la'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the greedy-score difficulty measure is defined independently of model outputs, and no prediction is reduced to a fitted input or a self-citation chain.

full rationale

The paper is an empirical benchmark study rather than a derivation, and its central claims are not circular by construction. The explanatory variable used to define 'difficult problem types' — the greedy score g — is computed by running a randomized greedy coloring algorithm 10,000 times per graph (Sections 2.2 and Appendix A). This quantity is a property of the graph and the algorithm, and is in no way a function of the LLM responses whose error rates it is used to stratify. The thresholds g ≥ 0.9, 0.5 ≤ g < 0.9, and g < 0.5 are stated a priori in Section 2.2, and model outputs are obtained independently via API calls with fixed prompts and scored by a separate parser. No parameter is fitted to the target error-rate claims, and the apparent correlation between low g and higher LLM error is an empirical finding, not a tautology. The paper even reports a result that is 'puzzling if we model the LLMs as approaching problems greedily but otherwise accurately' (Section 3.1), explicitly attributing part of the error increase to 'some other form of difficulty increase correlated with g substantially below 1'; this admission is inconsistent with a circular construction in which g is assumed to be identical to LLM difficulty. The genuine weaknesses of the paper — that g may not transfer as a difficulty measure to LLMs, that low-g problems correlate with other graph properties such as edge count, and that LRMs were evaluated on edge-selected subsets — are threats to construct validity and confound control, not cases where a claim reduces to its own input. There is also no load-bearing self-citation chain: the citations are to external prior work and are used for framing or comparison, not to define the benchmark or to justify its outcome. Accordingly, no specific circular step can be exhibited with a quoted reduction, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its free parameters are the chosen difficulty thresholds and the edge-count ranges for the LRM subset. The main axiomatic assumption is that the custom greedy score measures what makes problems hard for LLMs. The paper's claims depend on this measure being meaningful, since all difficulty-trend analysis uses it.

free parameters (2)

Difficulty thresholds on greedy score = 0.9 and 0.5
Thresholds chosen by hand to divide problems into easy, medium, and hard. The paper does not justify these specific values; different thresholds could change the difficulty categories.
Edge-count exclusions for LRM problem sets = 5v2c: edges 3-6; 6v3c: edges 7-12; 7v3c: edges 7-15; 8v4c: edges 14-23
The paper excludes low and high edge counts from LRM testing due to cost. These ranges define the 'interesting' region but are chosen post hoc based on where colorability varies, and they differ from the full problem set used for standard LLMs.

assumptions (2)

domain assumption The randomized greedy algorithm's success rate (g) across 10,000 trials is a meaningful proxy for problem difficulty for LLMs.
Invoked in Section 2.2 to categorize problems. The paper notes the algorithm is deliberately imperfect, but assumes its failure rate tracks the difficulty of finding a coloring, which is plausible but not proven.
domain assumption LLM outputs are correctly parsed and evaluated leniently, with manual review of ambiguous responses.
Described in Section 2.5. The evaluation is lenient and manual review is used, but the reliability of the parser is not quantified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring." pith.science (2026). https://pith.science/paper/XDDEJFNS

@misc{pith2026250207087,
  author       = {Pith},
  title        = {Pith review of: Evaluating the Systematic Reasoning Abilities of Large Language Models through Graph Coloring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDDEJFNS}},
  note         = {Machine review of arXiv:2502.07087}
}

abstract

Contemporary large language models are powerful problem-solving tools, but they exhibit weaknesses in their reasoning abilities which ongoing research seeks to mitigate. We investigate graph coloring as a means of evaluating an LLM's capacities for systematic step-by-step reasoning and possibility space exploration, as well as effects of semantic problem framing. We test Claude 3.5 Sonnet, Llama 3.1 405B, Gemini 1.5 Pro, GPT-4o, o1-mini, and DeepSeek-R1 on a dataset of $k$-coloring problems with $2 \leq k \leq 4$ and vertex count $4 \leq n \leq 8$, using partial algorithmic solvers to further categorize problems by difficulty. In addition to substantial but varying framing effects, we find that all models except o1-mini and R1 exhibit $>60\%$ error rates on difficult problem types in all frames ($>15\%$ for o1-mini and $>10\%$ for R1), and no model achieves perfect accuracy even in the simple domain of 2-coloring 4-vertex graphs. Our results highlight both the considerable recent progress in LLM systematic reasoning and the limits of its reliability, especially in relation to increasing computational costs. We expect that more complex graph coloring problems, and procedural generation of arbitrary-complexity reasoning problems more broadly, offer further untapped potential for LLM benchmarking.

Figures

Figures reproduced from arXiv: 2502.07087 by the authors.

**Figure 1.** Left: A simple undirected graph with 4 vertices and 4 edges. Center: A valid 2-coloring of the graph. Right: A different 4-vertex, 4-edge graph that cannot be 2-colored. ing n and k: one with 4-vertex graphs that must be 2- colored (“4v2c” for short), followed in increasing complexity by 5v2c, 6v3c, 7v3c, and 8v4c. 4v2c consists of all 2 C(4,2) = 64 possible 4-vertex graphs, minus the graph with no edges. Similarly… view at source ↗

**Figure 2.** Error rate for each standard LLM, problem set, problem type, and frame. The lighter-colored segment of each bar is the space between the lowest-error and highest-error frames. ror on 8v4c 0.5 ≤ g < 0.9 and g < 0.5, 50% error on 7v3c 0.5 ≤ g < 0.9 and g < 0.5, and 40% error on 6v3c 0.5 ≤ g < 0.9, in all four frames. Additionally, no combination of model and frame reaches 0% error on the combined set of 4v2c problems… view at source ↗

**Figure 3.** Error rate for each LRM, edge-selected problem set, problem type, and frame. The lighter-colored segment of each bar is the space between the errors of the two tested frames. The y-axis is compressed in comparison to [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 14 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Claude 3.5 Sonnet , 2024

Anthropic. Claude 3.5 Sonnet , 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

work page 2024
[3]

GPT-4 can't reason

Arkoudas, K. GPT-4 can't reason. arXiv preprint arXiv:2308.03762, 2023

arXiv 2023
[4]

DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning, 2025 a

DeepSeek-AI. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning, 2025 a . URL https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf

work page 2025
[5]

DeepSeek-R1 model card, 2025 b

DeepSeek-AI. DeepSeek-R1 model card, 2025 b . URL https://huggingface.co/deepseek-ai/DeepSeek-R1

work page 2025
[6]

L., Jiang, L., Lin, B

Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024

work page 2024
[7]

Kimi k1.5 : Scaling reinforcement learning with LLMs , 2025

Kimi Team . Kimi k1.5 : Scaling reinforcement learning with LLMs , 2025. URL https://github.com/MoonshotAI/Kimi-k1.5/blob/main/Kimi_k1.5.pdf

work page 2025
[8]

K., Dasgupta, I., Chan, S

Lampinen, A. K., Dasgupta, I., Chan, S. C., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models, like humans, show content effects on reasoning tasks. PNAS nexus, 3 0 (7): 0 pgae233, 2024

work page 2024

Show all 23 references

[9]

Introducing Llama 3.1 : Our most capable models to date, 2024

Meta. Introducing Llama 3.1 : Our most capable models to date, 2024. URL https://ai.meta.com/blog/meta-llama-3-1/

2024
[10]

GSM-Symbolic : Understanding the limitations of mathematical reasoning in large language models

Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. GSM-Symbolic : Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

2024 arXiv
[11]

U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A

Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435v10, 2024

2024 arXiv
[12]

Hello GPT-4o , 2024 a

OpenAI. Hello GPT-4o , 2024 a . URL https://openai.com/index/hello-gpt-4o/

2024
[13]

Learning to reason with LLMs , 2024 b

OpenAI. Learning to reason with LLMs , 2024 b . URL https://openai.com/index/learning-to-reason-with-llms/

2024
[14]

OpenAI o1-mini , 2024 c

OpenAI. OpenAI o1-mini , 2024 c . URL https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/

2024
[15]

and Hassabis, D

Pichai, S. and Hassabis, D. Our next-generation model: Gemini 1.5 , 2024. URL https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024

2024
[16]

Measuring and narrowing the compositionality gap in language models

Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., and Lewis, M. Measuring and narrowing the compositionality gap in language models. Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 5687--5711, 2023

2023
[17]

and Morstatter, F

Salinas, A. and Morstatter, F. The butterfly effect of altering prompts: How small changes and jailbreaks affect large language model performance. arXiv preprint arXiv:2401.03729, 2024

2024 arXiv
[18]

H., Sch\" a rli, N., and Zhou, D

Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Sch\" a rli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 4...

2023
[19]

On the self-verification limitations of large language models on reasoning and planning tasks

Stechly, K., Valmeekam, K., and Kambhampati, S. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08115, 2024

2024 arXiv
[20]

LLMs still can't plan; can LRMs ? a preliminary evaluation of OpenAI 's o1 on PlanBench

Valmeekam, K., Stechly, K., and Kambhampati, S. LLMs still can't plan; can LRMs ? a preliminary evaluation of OpenAI 's o1 on PlanBench . arXiv preprint arXiv:2409.13373, 2024

2024 arXiv
[21]

Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization

Wang, B., Yue, X., Su, Y., and Sun, H. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405.15071, 2024

2024 arXiv
[22]

Tree of thoughts: Deliberate problem solving with large language models

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024

2024
[23]

Larger and more instructable language models become less reliable

Zhou, L., Schellaert, W., Mart \' nez-Plumed, F., Moros-Daval, Y., Ferri, C., and Hern \'a ndez-Orallo, J. Larger and more instructable language models become less reliable. Nature, pp.\ 1--8, 2024

2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Claude 3.5 Sonnet , 2024

Anthropic. Claude 3.5 Sonnet , 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

work page 2024

[3] [3]

GPT-4 can't reason

Arkoudas, K. GPT-4 can't reason. arXiv preprint arXiv:2308.03762, 2023

arXiv 2023

[4] [4]

DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning, 2025 a

DeepSeek-AI. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning, 2025 a . URL https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSeek_R1.pdf

work page 2025

[5] [5]

DeepSeek-R1 model card, 2025 b

DeepSeek-AI. DeepSeek-R1 model card, 2025 b . URL https://huggingface.co/deepseek-ai/DeepSeek-R1

work page 2025

[6] [6]

L., Jiang, L., Lin, B

Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Le Bras, R., et al. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024

work page 2024

[7] [7]

Kimi k1.5 : Scaling reinforcement learning with LLMs , 2025

Kimi Team . Kimi k1.5 : Scaling reinforcement learning with LLMs , 2025. URL https://github.com/MoonshotAI/Kimi-k1.5/blob/main/Kimi_k1.5.pdf

work page 2025

[8] [8]

K., Dasgupta, I., Chan, S

Lampinen, A. K., Dasgupta, I., Chan, S. C., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models, like humans, show content effects on reasoning tasks. PNAS nexus, 3 0 (7): 0 pgae233, 2024

work page 2024