Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ExeRScope ties LLM code-reasoning failures to program structure, complexity, loops, and variable types.

desk verdict ExeRScope is a genuinely useful open-source tool for property-level analysis of code execution reasoning, but the paper's empirical claims rest on an OR-aggregation rule that confounds framework coverage with program properties; per-framework re-analysis is needed. read the letter →

arxiv 2501.18482 v1 pith:B7UBF3MU submitted 2025-01-30 cs.SE

Changshu Liu , Reyhaneh Jabbarvand This is my paper · ORCID
classification cs.SE
keywords codeexecutionreasoninglargelanguagemodelsprogramanalysisstaticanddynamiccomplexityLLMevaluationbenchmarkExeRScope
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models are increasingly evaluated on code execution reasoning: given a program and input, can the model predict the output or intermediate states? Existing frameworks report accuracy on fixed benchmarks but do not say which code properties make reasoning hard. ExeRScope is a tool that attaches each program to a set of static and dynamic properties — program constructs, cyclomatic and cognitive complexity, loop and recursion lengths, and variable types — and then relates those properties to whether an LLM predicted correctly. Applied to output predictions of six LLMs over 1,450 Python programs from four benchmarks, it yields a consistent picture: recursive and nested constructs, higher complexity, longer loops, and non-primitive types are all associated with lower accuracy. The point of the tool is to let such observations generalize along property axes rather than requiring a new benchmark for every new dataset.

What carries the argument

The central machinery is the property extraction and aggregation pipeline inside ExeRScope. Static analysis built on Python's `ast` module labels constructs; a control-flow graph built with PyCFG supplies cyclomatic and cognitive complexity; the `trace` module records runtime loop lengths and recursion lengths; and variable types are grouped into seven categories. The Visualizer then computes per-cluster accuracy and Spearman rank correlations, turning raw right/wrong predictions into a diagnostic report that attributes failures to code properties.

What would settle it

Run a controlled set of programs that differ only in loop length — same constructs, same complexity meter, and same output type — and check whether accuracy drops monotonically with iteration count; if it does not, the negative correlation reported here is a property of benchmark composition rather than a general reasoning limitation.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that code execution reasoning results can be made interpretable by a property-level analysis layer. ExeRScope tags each program with at least one construct label (basic, if, for, match, try, while, nested if, nested loop), measures complexity via lines of code, cyclomatic complexity, and cognitive complexity, traces execution to count loop iterations and recursion depth for each input, and classifies output variables into seven type categories. Aggregating predictions by these properties shows that LLMs handle conditionals better than loops, that complexity and loop length have a moderate-to-strong negative Spearman correlation with correct output prediction, and that models achieve high type-match but lower value-match on tuples, lists, and decimals. The paper presents these as empirical insights from six models on 1,450 programs, and as evidence that the tool can guide the design of benchmarks and instruction-tuning data.

Load-bearing premise

The load-bearing premise is that correct output prediction — counted as correct if any of the three evaluation frameworks produced the right answer — faithfully measures how well an LLM reasons about code execution, property by property.

Editorial extensions

If this is right

  • Benchmark designers can deliberately include recursive and nested constructs, non-primitive types, and longer loops, since these are the conditions where models measurably struggle.
  • Model developers can use ExeRScope's per-property reports to choose instruction-tuning targets — for instance, SemCoder, which performs well on CRUXEval, still degrades on longer loops and complex types.
  • Existing execution-reasoning results can be re-analyzed across new datasets by matching property distributions, without generating a new benchmark from scratch.
  • The same pipeline can be applied to Java programs and to arbitrary programs together with execution reasoning results, so the analysis is not tied to the four studied benchmarks.
  • Observations about LLM execution reasoning become property-based rather than dataset-based, making claims easier to test and transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The property lens suggests a testable 'iteration budget' hypothesis: LLM execution reasoning may fail because errors compound per loop iteration or recursion step; a controlled experiment holding all other properties fixed while scaling only iteration count would isolate this effect.
  • The same static-dynamic property profile could be used to predict performance on unseen programs before running a model, turning benchmark difficulty into a property vector rather than a dataset identity.
  • If the findings generalize, they point to a concrete data-generation strategy: synthesize minimal programs that each instantiate a single hard property (e.g., nested loops with many iterations) for targeted fine-tuning or probing.
  • The type-match versus value-match split suggests that LLMs may learn 'type-shaped' templates without tracking the internal structure of complex values; a follow-up could test this by permuting the internal layout of lists and tuples while keeping types the same.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ExeRScope, a tool that combines static and dynamic program analysis with statistical and visualization utilities to analyze the results of code execution reasoning frameworks for LLMs. It extracts program constructs, complexity metrics, dynamic properties (e.g., loop lengths), and variable types from Python programs, and relates these properties to LLM prediction correctness. The tool is demonstrated on output prediction results from six LLMs over 1450 Python programs from four benchmarks (Avatar, ClassEval, CRUXEval, HumanEval), drawn from three existing frameworks. The paper reports that recursive and nested constructs, higher complexity, longer loops, and non-primitive types are negatively associated with LLM code execution reasoning performance, and argues that these findings motivate richer benchmarks and better instruction tuning.

Significance. If the empirical trends are reliable, ExeRScope fills a real gap: it provides a systematic, property-level diagnostic for code execution reasoning beyond raw accuracy numbers. The artifact is publicly available, and the approach is transparent and reproducible in principle. The central claims, however, rest on a post-hoc aggregation of overlapping framework results and on correlations reported without uncertainty quantification; these load-bearing issues must be fixed before the empirical conclusions can be accepted.

major comments (3)
  1. [Section 4, first paragraph] The stated aggregation rule, "when the programs overlap between any of the three studied frameworks, we chose the correct prediction if it existed," gives each program as many chances to be marked correct as the number of frameworks that evaluated it. Coverage is not balanced across the four benchmarks: CRUXEval appears to be evaluated only by CRUXEval while HumanEval programs are covered by multiple frameworks, so a HumanEval program can be scored correct if any of several independent predictions is correct. Because complexity and construct distributions differ across benchmarks, the Spearman correlations in Figures 3-4 and the construct-cluster percentages in Figure 2 may reflect this scoring asymmetry rather than LLM reasoning ability. Please re-run the property-level analyses separately for each framework (or restrict to programs evaluated by all frameworks), or provide evidence that the OR-aggregation does not bias the property trends.
  2. [Sections 4.2 and 4.3, Figures 3-4] The manuscript reports Spearman's rho values and labels them "moderate to strong" without p-values, confidence intervals, or the number of programs in each cluster. With six LLMs and uneven cluster sizes, these correlations are not statistically characterized. Add significance tests (e.g., permutation or bootstrap) and report sample sizes and CIs for each correlation.
  3. [Figures 2-5] In the version of the manuscript provided for review, the figure images are absent; only captions appear. The empirical results that support the central claims are presented exclusively in these figures, so the evaluation cannot be verified. The authors must ensure the figures are embedded and legible in the submission, and ideally provide the underlying data tables.
minor comments (5)
  1. [Page header, Section 1] The page header reads "June 23–275, 2025"; this should be "June 23–27, 2025."
  2. [Section 3.1.4] The "Object" variable-type example is shown as a Python dict literal (e.g., {"age":10, "gender":"female"}); clarify how dicts are distinguished from the List and Tuple categories and whether Object subsumes compound built-in types.
  3. [Section 4.1] The phrase "recursions (while loops and for loops)" is confusing: while and for loops are iterative constructs, not recursion. The intended contrast appears to be between conditional constructs and loop/recursive constructs; please reword for precision.
  4. [Section 3.1.2] Cognitive complexity is listed as a supported program complexity metric, but no definition or implementation reference is given beyond the mention that lizard is not used. Add a precise definition or cite the standard definition of cognitive complexity.
  5. [Figures 3 and 4 captions] The figure captions do not state what the x-axis and y-axis represent or how the clusters (bars) are formed. Add explicit axis labels and a note on the clustering of complexity values.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ExeRScope reports descriptive statistics over externally produced prediction results; no fitted parameter or self-citation chain is load-bearing.

full rationale

The paper's analytic chain is: take existing code execution reasoning predictions from CodeMind, REval, and CRUXEval; tag the subject programs with static and dynamic program properties; and then compute correctness rates, Spearman correlations, and cluster percentages relating those properties to success. No step defines a program property in terms of the prediction outcome, and no parameter is fitted to a subset of data and then renamed as a prediction. The OR-aggregation rule stated at the start of Section 4 ('when the programs overlap between any of the three studied frameworks, we chose the correct prediction if it existed') is a measurement-construction choice that could bias benchmark comparisons by giving multi-framework programs extra chances to count as correct; this is a validity or confound concern, not a circularity, because the underlying predictions still come from external frameworks and the aggregation does not make the reported property-outcome relationships true by construction. The references to prior frameworks are external evidence, and no uniqueness theorem or self-citation is invoked to force the conclusions. The central empirical findings are independent descriptive observations rather than reduced, by definition or by fit, to their own inputs, so the correct circularity score is 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the accuracy and representativeness of the program property extraction, the statistical measures used, and the validity of output prediction as a proxy for code execution reasoning. No fitted parameters are introduced; the analysis is based on existing evaluation results.

assumptions (6)
  • domain assumption The Python ast module correctly identifies the program constructs and nesting structures used for tagging.
    ExeRScope tags each program with constructs via Python's ast module (Section 3.1.1).
  • domain assumption Cyclomatic complexity (CC = E - N + 2P) and cognitive complexity, computed via PyCFG, are meaningful proxies for how hard a program is for an LLM to execute mentally.
    Section 3.1.2 defines these metrics and uses them as predictors.
  • domain assumption Dynamic execution with the provided inputs yields representative loop lengths and recursion lengths.
    Section 3.1.3 uses the trace library to collect loop and recursion lengths per input.
  • standard math Spearman rank correlation is an appropriate measure of association between program properties and prediction success.
    Section 4.2 uses Spearman's rho to test complexity versus accuracy.
  • domain assumption Output prediction is a valid proxy for code execution reasoning; the evaluation is based solely on output prediction.
    The paper states 'Our evaluation of ExeRScope is based on output prediction since it is the only common execution reasoning task in existing frameworks.'
  • ad hoc to paper Aggregating overlapping results by choosing the correct prediction when any framework produced it is unbiased for the property-level analysis.
    Section 4, first paragraph: 'when the programs overlap between any of the three studied frameworks, we chose the correct prediction if it existed.'

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models." pith.science (2026). https://pith.science/paper/B7UBF3MU

@misc{pith2026250118482,
  author       = {Pith},
  title        = {Pith review of: A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7UBF3MU}},
  note         = {Machine review of arXiv:2501.18482}
}
read the original abstract

Code Executing Reasoning is becoming a new non-functional metric that assesses the ability of large language models (LLMs) in programming tasks. State-of-the-art frameworks (CodeMind or REval) and benchmarks (CruxEval) usually focus on LLM's prediction of a given code's input/output or intermediate variable states/values on limited programs. However, there is no tool for more in-depth analysis of the results. Without such a tool, the observations about LLM's code execution reasoning cannot be generalized to more datasets, preventing the research community and practitioners from devising the next generation of LLMs with better code execution reasoning abilities. This paper introduces ExeRScope, a series of tools and heuristics to analyze the result of code execution reasoning frameworks to understand better the impact of code properties in the studied benchmarks on the code execution reasoning. With such tooling, analysis can be generalized to code with similar properties without the urgent need to design more benchmarks, which is a cumbersome effort.

Figures

Figures reproduced from arXiv: 2501.18482 by the authors.

Figure 1
Figure 1. Overview of ExeRScope. code execution reasoning results (i.e., variable value predictions and whether the LLM correctly predicted the value or not), ExeRScope extracts specific properties (program constructs, code complexity, dynamic properties such as recursion length, and output type) and investigates how these factors vary in correct predictions versus incorrect ones. It then plots the results with appropriate vi… view at source ↗
Figure 2
Figure 2. Impact of different programming constructs on code ex [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Impact of cyclomatic complexity on execution reasoning [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of Loop Length on execution reasoning of LLMs. The [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ATLAS: Multi-View Code Representation Tool for C and C++ Source Programs

    cs.SE 2025-12 conditional novelty 5.0 of 10

    ATLAS claims no-build aligned AST/CFG/DFG extraction for C/C++ at 96.8%/91.7% CFG correctness, and a 34.7-point LLM line-coverage lift when fed its CFG paths.

Reference graph

Works this paper leans on

39 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Matplot library for visualization

    2025. Matplot library for visualization. https://matplotlib.org/

    work page 2025

  2. [2]

    Network Analysis in Python using NetworkX

    2025. Network Analysis in Python using NetworkX. https://networkx.org/

    work page 2025

  3. [3]

    PyGraphVis library for visualization

    2025. PyGraphVis library for visualization. https://pygraphviz.github.io/

    work page 2025

  4. [4]

    Python AST module

    2025. Python AST module. https://docs.python.org/3/library/ast.html

    work page 2025

  5. [6]

    Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation. arXiv preprint arXiv:2108.11590 (2021)

    arXiv 2021

  6. [7]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al . 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv preprint arXiv:2401.02954 (2024)

    arXiv 2024

  7. [8]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)

    arXiv 2023

  8. [9]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning Runtime Behavior of a Program with LLM: How Far Are We?. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 140–152

    work page 2024

Show all 39 references
  1. [10]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021 arXiv

  2. [11]

    Rohan Deshpande, Jerry Chen, and Isabelle Lee. 2021. RecT: A Recursive Trans- former Architecture for Generalizable Mathematical Reasoning.. In NeSy. 165– 175

    2021

  3. [12]

    Yangruibo Ding, Jinjun Peng, Marcus J Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. SemCoder: Training Code Language Models with Compre- hensive Semantics. arXiv preprint arXiv:2406.01006 (2024)

    2024 arXiv

  4. [13]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engin...

    2024

  5. [14]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Syn- naeve, and Sida I Wang. 2024. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. arXiv preprint arXiv:2401.03065 (2024)

    2024 arXiv

  6. [15]

    Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. 2023. Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning. arXiv preprint arXiv:2312.10160 (2023)

    2023 arXiv

  7. [16]

    Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathprompter: Mathe- matical reasoning using large language models. arXiv preprint arXiv:2303.05398 (2023)

    2023 arXiv

  8. [17]

    Katie Kang, Amrith Setlur, Dibya Ghosh, Jacob Steinhardt, Claire Tomlin, Sergey Levine, and Aviral Kumar. 2024. What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? arXiv preprint arXiv:2411.07681 (2024)

    2024 arXiv

  9. [18]

    Emanuele La Malfa, Christoph Weinhuber, Orazio Torre, Fangru Lin, Anthony Cohn, Nigel Shadbolt, and Michael Wooldridge. 2024. Code Simulation Challenges for Large Language Models. arXiv preprint arXiv:2401.09074 (2024)

    2024 arXiv

  10. [19]

    Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. 2023. Code Execution with Pre- trained Language Models. arXiv preprint arXiv:2305.05383 (2023)

    2023 arXiv

  11. [20]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173 (2024)

    2024 arXiv

  12. [21]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583 (2023)

    2023 arXiv

  13. [22]

    Antonio Valerio Miceli-Barone, Fazl Barez, Ioannis Konstas, and Shay B Cohen

  14. [23]

    Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. 2023. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain.arXiv preprint arXiv:2310.14053 (2023)

    2023 arXiv

  15. [24]

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al. 2024. Granite code models: A family of open foundation models for code intelligence. arXiv preprint arXiv:...

    2024 arXiv

  16. [25]

    Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. NExT: Teaching Large Language Models to Reason about Code Execution. arXiv preprint arXiv:2404.14662 (2024)

    2024 arXiv

  17. [26]

    OpenAI. 2023. GPT-4 Technical Report. https://arxiv.org/abs/2303.08774 (2023)

    2023 arXiv

  18. [27]

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

    2024 arXiv

  19. [28]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  20. [29]

    Charles Spearman. 1961. The proof and measurement of association between two things. (1961)

    1961

  21. [30]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  22. [31]

    Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambham- pati. 2022. Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). arXiv preprint arXiv:2206.10498 (2022)

    2022 arXiv

  23. [32]

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731 (2023)

    2023 arXiv

  24. [33]

    ExeRScope Tool Artifact Website. 2025. https://github.com/Intelligent-CAT- Lab/ExeRScope

    2025

  25. [34]

    Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2023. Reasoning or reciting? explor- ing the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477 (2023)

    2023 arXiv

  26. [35]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)

    2024 arXiv

  27. [36]

    Terry Yin. 2025. Lizard. https://github.com/terryyin/lizard

    2025

  28. [37]

    Dylan Zhang, Curt Tigges, Zory Zhang, Stella Biderman, Maxim Raginsky, and Talia Ringer. 2024. Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion. arXiv preprint arXiv:2401.12947 (2024)

    2024 arXiv

  29. [38]

    Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. 2024. Can LLM Graph Reasoning Generalize beyond Pattern Memorization? arXiv preprint arXiv:2406.15992 (2024)

    2024 arXiv

  30. [39]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931 (2024)

    2024 arXiv

  31. [2023]

    arXiv preprint arXiv:2305.15507 (2023)

    The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python. arXiv preprint arXiv:2305.15507 (2023)

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.