pith. machine review for the scientific record. sign in

arxiv: 2512.14917 · v3 · submitted 2025-12-16 · 💻 cs.SE

Recognition: no theorem link

Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings

Changshu Liu , Alireza Ghazanfari , Yang Chen , Reyhaneh Jabbarvand
Authors on Pith no claims yet

Pith reviewed 2026-05-16 21:20 UTC · model grok-4.3

classification 💻 cs.SE
keywords code reasoningLLM evaluationcode complexity metricsreal-world codeprogram analysisbenchmark datasetPython repositoriesLLM assessment
0
0 comments X

The pith

Most existing code-reasoning benchmarks for LLMs test only lower-complexity problems, missing real-world complexities.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper constructs a dataset of 1200 reasoning problems from existing benchmarks and popular GitHub Python repositories. Using static and dynamic analysis, it handles complex types beyond primitives. A majority-vote over nine complexity metrics separates the problems into Lower Complexity (LC) and Higher Complexity (HC) groups. This reveals that existing benchmarks are mostly LC and do not represent real-world code complexity.

Core claim

The authors construct a dataset of 1200 code reasoning problems and categorize them as Lower Complexity (LC) or Higher Complexity (HC) using a majority vote over nine code complexity metrics. Their analysis shows that problems from existing benchmarks mostly belong to the LC category, while those from GitHub repositories better capture real-world complexities such as inter-procedural dependencies and non-primitive types.

What carries the argument

Majority-vote categorization over nine code-complexity metrics to divide problems into Lower Complexity (LC) and Higher Complexity (HC) groups.

If this is right

  • Evaluations relying on existing benchmarks will likely underestimate LLM difficulties with real code dependencies and custom types.
  • The new dataset enables testing models on problems that include inter-procedural calls and non-primitive types.
  • Models that perform well on LC tasks may still fail when presented with HC problems drawn from actual repositories.
  • Benchmark designers should add more HC problems to better reflect practical code reasoning demands.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Reported LLM code-reasoning scores on current benchmarks may overstate readiness for large-scale software projects.
  • The LC/HC split could guide the creation of staged training sets that increase complexity gradually.
  • Similar metric-based categorization might apply to evaluating code generation or repair tasks beyond pure reasoning.
  • Practitioners integrating LLMs into codebases should expect performance drops on tasks involving deep nesting or API chains.

Load-bearing premise

That a majority vote over the nine chosen code-complexity metrics produces a semantically meaningful and stable separation between lower- and higher-complexity problems.

What would settle it

Re-running the nine metrics on a fresh sample of GitHub repositories where existing benchmark problems no longer cluster predominantly in the LC category would undermine the claimed separation.

Figures

Figures reproduced from arXiv: 2512.14917 by Alireza Ghazanfari, Changshu Liu, Reyhaneh Jabbarvand, Yang Chen.

Figure 1
Figure 1. Figure 1: A real-world reasoning problem (highlighted methods) For a systematic comparison of LLMs’ code reasoning performance on real-world programs and those used in code reasoning studies, we constructed a dataset of 1,200 reasoning problems (details in §2) from existing code rea￾soning evaluation benchmarks (Avatar, ClassEval, CRUX￾Eval, and HumanEval), as well as real-world projects (from widely-used SWE-bench … view at source ↗
Figure 2
Figure 2. Figure 2: Complexity distribution of reasoning problems from prior techniques, compared to real-world programs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of custom types in real-world reasoning problems and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Breakdown of the reasoning problems across complexity levels and metrics for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: RE2-Bench and RE2-Bench-lite overview The Silhouette score measures how well each rea￾soning problem fits within its complexity category, based on the distance to neighboring points within versus outside its cluster (local reliability). DBI mea￾sures the average similarity between two complex￾ity categories, based on intra-cluster dispersion and inter-cluster distance (global soundness). Using both ensures… view at source ↗
Figure 6
Figure 6. Figure 6: Prompt template for Input and Output Prediction [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of the correct (solid) and incorrect (dotted) predictions ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Unique and common problems each LLM succeeds in predicting their inputs and outputs. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: False negative categorization per source [PITH_FULL_IMAGE:figures/full_fig_p011_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: False negative in Avatar [51, 52, 54, 56] [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Impact of programming constructs on input and output prediction. Abbreviations are: Basic (B), For [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: illustrates another example from a real-world project: the method split_super_sub takes text as input. The loop partitions the input into the base name, superscript (supers), and subscript (subs). After the loop, an if-condition checks whether the name ends with digits. If so, the number is placed at the beginning of the subscript. In this example, Gemini-3-Pro and GPT-5-mini predict "alpha1^+", whereas D… view at source ↗
Figure 13
Figure 13. Figure 13: Comparison of the call chain size distribution between successful ( [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Comparison between the performance of reasoning and low-reasoning/non-reasoning LLMs [PITH_FULL_IMAGE:figures/full_fig_p013_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Categorization of reasoning failures Claude-Haiku -4.5 DeepSeek -V3.2 GPT-5 -mini Input Prediction Output Prediction Conditional Statements Recursions Structural Complexity Oversight Call Dependencies Types Gemini-3 -Pro CWM Claude-Haiku -4.5(RD) DeepSeek -V3.2 (RD) Gemini-3 -Pro(LR) GPT-5 -mini(RD) CWM -Pretrain [PITH_FULL_IMAGE:figures/full_fig_p014_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Breakdown of the reasoning failure categorization per individual LLM [PITH_FULL_IMAGE:figures/full_fig_p014_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Examples of reasoning failures and their failure categories [ [PITH_FULL_IMAGE:figures/full_fig_p015_17.png] view at source ↗
read the original abstract

Code reasoning tasks are becoming prevalent in large language model (LLM) assessments. Yet, there is a dearth of studies on the impact of real-world complexities on code reasoning, e.g., inter- or intra-procedural dependencies, API calls, deeply nested constructs, and non-primitive complex types. Evaluating LLMs under such a simplistic setting poses a significant threat to assumptions about their generalizability in practice. To enable a more realistic evaluation of code reasoning, we construct a dataset of 1200 reasoning problems from two sources: existing code reasoning benchmarks and popular GitHub Python repositories. Our pipeline leverages static and dynamic program analysis to automatically serialize/deserialize compound, complex, and custom types galore in real-world code, going far beyond only primitive types used in prior studies. A key feature of our dataset is categorizing each reasoning problem as Lower Complexity (LC) or Higher Complexity (HC) via a principled majority-vote mechanism over nine diverse and interpretable code-complexity metrics, yielding two well-separated, semantically meaningful categories of problem difficulty suitable for precise calibration of LLM reasoning ability. This categorization shows that the problems used in existing code-reasoning evaluation mostly belong to the LC category, failing to represent real-world complexity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper constructs a dataset of 1200 code-reasoning problems drawn from existing benchmarks and GitHub Python repositories. It applies static and dynamic program analysis to serialize complex and custom types, then assigns each problem to Lower Complexity (LC) or Higher Complexity (HC) via majority vote across nine code-complexity metrics. The central claim is that this produces two well-separated categories and that existing benchmarks fall overwhelmingly into the LC category, thereby failing to capture real-world inter-procedural dependencies and non-primitive types.

Significance. If the majority-vote split is shown to be stable and to align with actual LLM failure modes, the work would supply a concrete, reproducible method for calibrating code-reasoning evaluations against realistic complexity, directly addressing the generalizability gap highlighted in the abstract.

major comments (2)
  1. [Categorization section (majority-vote mechanism)] The abstract and the section describing the categorization pipeline supply no quantitative validation of the majority-vote threshold, no inter-metric agreement statistics, and no sensitivity analysis; without these, the assertion that the procedure yields 'two well-separated, semantically meaningful categories' remains unsupported and load-bearing for the claim that existing benchmarks are mostly LC.
  2. [Evaluation / Results section] No correlation is reported between the LC/HC labels and actual LLM reasoning performance (e.g., pass rates or error patterns on the 1200 problems); this external validation is required to confirm that the nine-metric vote captures the difficulties the paper attributes to real-world code.
minor comments (1)
  1. [Dataset construction] The nine metrics are described as 'diverse and interpretable' but are not enumerated or tabulated; a concise table listing each metric, its static/dynamic nature, and its weighting in the vote would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We have strengthened the manuscript by adding quantitative validation for the categorization procedure and preliminary external validation against LLM performance. Point-by-point responses follow.

read point-by-point responses

  1. Referee: [Categorization section (majority-vote mechanism)] The abstract and the section describing the categorization pipeline supply no quantitative validation of the majority-vote threshold, no inter-metric agreement statistics, and no sensitivity analysis; without these, the assertion that the procedure yields 'two well-separated, semantically meaningful categories' remains unsupported and load-bearing for the claim that existing benchmarks are mostly LC.

    Authors: We agree that explicit quantitative support for the majority-vote procedure strengthens the central claim. In the revised version we have added: (i) Fleiss' kappa of 0.71 across the nine metrics indicating substantial inter-metric agreement; (ii) sensitivity analysis showing that the LC/HC assignment is stable for majority thresholds between 5/9 and 7/9, with 84% of problems retaining the same label; and (iii) statistical separation tests (two-sample t-tests, all p < 0.001) on each metric between the resulting LC and HC groups. These additions directly support that the categories are well-separated and semantically meaningful. revision: yes

  2. Referee: [Evaluation / Results section] No correlation is reported between the LC/HC labels and actual LLM reasoning performance (e.g., pass rates or error patterns on the 1200 problems); this external validation is required to confirm that the nine-metric vote captures the difficulties the paper attributes to real-world code.

    Authors: We concur that demonstrating alignment between the LC/HC labels and LLM behavior provides important external validation. The revised manuscript now includes a new analysis on a representative sample of 300 problems (150 LC, 150 HC) evaluated with three contemporary models. Results show substantially lower pass@1 rates on HC problems (average 24% vs. 49% on LC) and error patterns dominated by failures on non-primitive types and inter-procedural dependencies, consistent with the difficulties highlighted in the paper. Full evaluation across all 1200 problems is noted as future work due to computational cost. revision: partial

Circularity Check

0 steps flagged

No circularity: LC/HC labels derived from independent metrics

full rationale

The paper's central categorization step applies a majority vote over nine standard static and dynamic code-complexity metrics (e.g., cyclomatic complexity, nesting depth, inter-procedural dependencies) to label problems as LC or HC. These metrics are defined externally via program analysis and are not derived from, fitted to, or defined in terms of LLM outputs, predictions, or the final claim about existing benchmarks. The observation that prior benchmarks fall mostly into LC follows directly from applying the labels to the collected problems; no equation reduces to itself, no parameter is renamed as a prediction, and no self-citation chain is required to justify the split. The derivation chain remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the nine chosen metrics plus majority vote capture real-world complexity and that the sampled GitHub repositories are representative; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption The nine code-complexity metrics are valid and sufficient indicators of practical reasoning difficulty
    Invoked when the majority-vote procedure is used to label problems as LC or HC

pith-pipeline@v0.9.0 · 5525 in / 1212 out tokens · 18686 ms · 2026-05-16T21:20:14.088968+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 32 canonical work pages · 8 internal anchors

  1. [1]

    Code Simulation as a Proxy for High-order Tasks in Large Language Models.arXiv preprint arXiv:2502.03568 (2025)

    2025. Code Simulation as a Proxy for High-order Tasks in Large Language Models.arXiv preprint arXiv:2502.03568 (2025)

    work page arXiv 2025

  2. [2]

    Hiralal Agrawal and Joseph R Horgan. 1990. Dynamic program slicing.ACM SIGPlan Notices25, 6 (1990), 246–256

    1990

  3. [3]

    Wasi Uddin Ahmad, Md Golam Rahman Tushar, Saikat Chakraborty, and Kai-Wei Chang. 2021. Avatar: A parallel corpus for java-python program translation.arXiv preprint arXiv:2108.11590(2021)

    work page arXiv 2021

  4. [4]

    Miltiadis Allamanis, Henry Jackson-Flux, and Marc Brockschmidt. 2021. Self-supervised bug detection and repair. Advances in Neural Information Processing Systems34 (2021), 27865–27876

    2021

  5. [5]

    2025.System Card: Claude Haiku 4.5

    Anthropic. 2025.System Card: Claude Haiku 4.5. Technical Report. Anthropic. https://www.anthropic.com/claude- haiku-4-5-system-card Accessed 2026-01-19

    2025

  6. [6]

    Claas Beger and Saikat Dutta. 2025. CoCoNUT: Structural Code Understanding does not fall out of a tree.arXiv preprint arXiv:2501.16456(2025)

    work page arXiv 2025

  7. [7]

    Gregory Bolet, Giorgis Georgakoudis, Konstantinos Parasyris, Harshitha Menon, Niranjan Hasabnis, Kirk W Cameron, and Gal Oren. 2025. Counting Without Running: Evaluating LLMs’ Reasoning About Code Complexity.arXiv preprint arXiv:2512.04355(2025)

    work page arXiv 2025

  8. [8]

    Dimitrios Stamatios Bouras, Yihan Dai, Tairan Wang, Yingfei Xiong, and Sergey Mechtaev. 2025. HoarePrompt: Structural Reasoning About Program Correctness in Natural Language.arXiv preprint arXiv:2503.19599(2025)

    work page arXiv 2025

  9. [9]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we?arXiv preprint arXiv:2403.16437(2024)

    work page arXiv 2024

  10. [10]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far Are We? . In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 140–152. doi:10.1109/ICSE55347.2025.00012

    work page doi:10.1109/icse55347.2025.00012 2025

  11. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  12. [12]

    Jade Copet, Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, et al. 2025. Cwm: An open-weights llm for research on code generation with world models. arXiv preprint arXiv:2510.02387(2025)

    work page arXiv 2025

  13. [13]

    David L Davies and Donald W Bouldin. 2009. A cluster separation measure.IEEE transactions on pattern analysis and machine intelligence2 (2009), 224–227

    2009

  14. [14]

    Edsger W Dijkstra and Edsger W Dijkstra. 1982. Program inversion.Selected Writings on Computing: A Personal Perspective(1982), 351–354

    1982

  15. [15]

    Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Animashree Anandkumar, et al. 2024. Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization.Advances in Neural Information Processing Systems37 (2024), 44323–44365

    2024

  16. [16]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability Detection with Code Language Models: How Far Are We? . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 469–481. doi:10.110...

    work page doi:10.1109/icse55347.2025.00038 2025

  17. [17]

    Yangruibo Ding, Marcus J Min, Gail Kaiser, and Baishakhi Ray. 2024. Cycle: Learning to self-refine the code generation. Proceedings of the ACM on Programming Languages8, OOPSLA1 (2024), 392–418

    2024

  18. [18]

    Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. Semcoder: Training code language models with comprehensive semantics reasoning.Advances in Neural Information Processing Systems37 (2024), 60275–60308

    2024

  19. [19]

    Peter Dinges and Gul Agha. 2014. Targeted test input generation using symbolic-concrete backward execution. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 31–36. , Vol. 1, No. 1, Article . Publication date: February 2018. 20 Changshu Liu, Alireza Ghazanfari, Yang Chen, Reyhaneh Jabbarvand

    2014

  20. [20]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. 2022. A survey on in-context learning.arXiv preprint arXiv:2301.00234(2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  21. [21]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861(2023)

    work page arXiv 2023

  22. [22]

    Dhruv Gautam, Spandan Garg, Jinu Jang, Neel Sundaresan, and Roshanak Zilouchian Moghaddam. 2025. Refactorbench: Evaluating stateful reasoning in language agents through code.arXiv preprint arXiv:2503.07832(2025)

    work page arXiv 2025

  23. [23]

    Aryo Pradipta Gema, Alexander Hägele, Runjin Chen, Andy Arditi, Jacob Goldman-Wetzler, Kit Fraser-Taliente, Henry Sleight, Linda Petrini, Julian Michael, Beatrice Alex, et al. 2025. Inverse scaling in test-time compute.arXiv preprint arXiv:2507.14417(2025)

    work page arXiv 2025

  24. [24]

    2025.Gemini 3 Pro

    Google. 2025.Gemini 3 Pro. https://deepmind.google/models/gemini/pro/

    2025

  25. [25]

    Google. 2025. Thinking of Gemini models. https://docs.cloud.google.com/vertex-ai/generative-ai/docs/thinking

    2025

  26. [26]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution.arXiv preprint arXiv:2401.03065(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Ashish Hooda, Mihai Christodorescu, Miltiadis Allamanis, Aaron Wilson, Kassem Fawaz, and Somesh Jha. 2024. Do large code models understand programming concepts? a black-box approach.arXiv preprint arXiv:2402.05980(2024)

    work page arXiv 2024

  28. [28]

    Ali Reza Ibrahimzada, Kaiyao Ke, Mrigank Pawagi, Muhammad Salman Abid, Rangeet Pan, Saurabh Sinha, and Rey- haneh Jabbarvand. 2024. Repository-level compositional code translation and validation.arXiv preprint arXiv:2410.24117 (2024)

    work page arXiv 2024

  29. [29]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Rob Kopel. 2025. EXecution-Eval:Can language models execute real-world code? (2025)

    2025

  31. [31]

    Diana Kornbrot. 2014. Point biserial correlation.Wiley StatsRef: Statistics Reference Online(2014)

    2014

  32. [32]

    Thanh Le-Cong, Bach Le, and Toby Murray. 2025. Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference.arXiv preprint arXiv:2503.04779(2025)

    work page arXiv 2025

  33. [33]

    Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. 2025. S*: Test time scaling for code generation.arXiv preprint arXiv:2502.14382(2025)

    work page arXiv 2025

  34. [34]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek-v3. 2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2024. CodeMind: Evaluating Large Language Models for Code Reasoning.arXiv preprint arXiv:2402.09664(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  36. [36]

    Changshu Liu, Yang Chen, and Reyhaneh Jabbarvand. 2025. Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models.arXiv preprint arXiv:2510.15079(2025)

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  37. [37]

    Changshu Liu and Reyhaneh Jabbarvand. 2025. A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models.arXiv preprint arXiv:2501.18482(2025)

    work page arXiv 2025

  38. [38]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems36 (2023), 21558–21572

    2023

  39. [39]

    Zhicun Lyu, Xinye Li, Zheng Xie, and Ming Li. 2025. Top Pass: improve code generation by pass@ k-maximized code ranking.Frontiers of Computer Science19, 8 (2025), 198341

    2025

  40. [40]

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al

  41. [41]

    Language models are few-shot learners.arXiv preprint arXiv:2005.141651 (2020), 3

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  42. [42]

    Thomas J McCabe. 1976. A complexity measure.IEEE Transactions on software Engineering4 (1976), 308–320

    1976

  43. [43]

    Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. 2023. Inverse scaling: When bigger isn’t better.arXiv preprint arXiv:2306.09479 (2023)

    work page arXiv 2023

  44. [44]

    OpenAI. 2025. GPT-5-mini. https://chat.openai.com/. https://chat.openai.com/

    2025

  45. [45]

    OpenRouter. 2025. Reasoning Tokens. https://openrouter.ai/docs/guides/best-practices/reasoning-tokens

    2025

  46. [46]

    Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2024. Lost in translation: A study of bugs introduced by large language models while translating code. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  47. [47]

    Saurabh Pujar, Ira Ceka, Irene Manotas, Gail Kaiser, Baishakhi Ray, and Shyam Ramji. 2025. Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action.arXiv e-prints(2025), arXiv–2506

    2025

  48. [48]

    RE2-Bench. 2026. Artifact Website of RE2-Bench. https://github.com/Intelligent-CAT-Lab/RE2-Bench. , Vol. 1, No. 1, Article . Publication date: February 2018. Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings 21

    2026

  49. [49]

    RE2-Bench. 2026. Example of Reasoning Failure under Incomplete Predicate Reasoning. https://github.com/Intelligent- CAT-Lab/RE2Bench/blob/main/results/output_prediction/gemini-3-pro-preview-reasoning/HC/sympy__sympy- 13177%40%40sympy.core.mod.py%40%40doit.txt

    2026

  50. [50]

    RE2-Bench. 2026. Example of Reasoning Failure under Incorrect Type Resolution. https://github.com/Intelligent- CAT-Lab/RE2Bench/blob/main/results/output_prediction/cwm-pretrain/HC/scikit-learn__scikit-learn- 25589%40%40sklearn.preprocessing._encoders.py%40%40_map_drop_idx_to_infrequent.txt#L23

    2026

  51. [51]

    RE2-Bench. 2026. Example of Reasoning Failure under Lost Iteration Tracking. https://github.com/Intelligent-CAT- Lab/RE2Bench/blob/main/results/output_prediction/claude-haiku-4.5-reasoning/HC/sympy%40%40sympy_polys_ factortools.py%40%40dmp_zz_wang_hensel_lifting_L939.txt

    2026

  52. [52]

    RE2-Bench. 2026. False Negative Detected from Claude-Haiku-4.5 in Avatar. https://github.com/Intelligent-CAT- Lab/RE2Bench/blob/main/results/input_prediction/deepseek-v3.2-reasoning/LC/codeforces_379_A.txt

    2026

  53. [53]

    RE2-Bench. 2026. False Negative Detected from DeepSeek-V3.2 in Avatar. https://github.com/Intelligent-CAT- Lab/RE2Bench/blob/main/results/input_prediction/deepseek-v3.2-reasoning/LC/codeforces_379_A.txt

    2026

  54. [54]

    RE2-Bench. 2026. False Negative Detected from DeepSeek-V3.2 in Real-world Projects. https://github.com/Intelligent- CAT-Lab/RE2Bench/blob/main/results/input_prediction/deepseek-v3.2-reasoning/HC/sympy__sympy- 11822%40%40sympy.printing.conventions.py%40%40split_super_sub.txt

    2026

  55. [55]

    RE2-Bench. 2026. False Negative Detected from Gemini-3-Pro in Avatar. http://github.com/Intelligent-CAT-Lab/ RE2Bench/blob/main/results/input_prediction/gemini-3-pro-preview-reasoning/LC/codeforces_379_A.txt

    2026

  56. [56]

    RE2-Bench. 2026. False Negative Detected from Gemini-3-Pro in Real-world Projects. https://github.com/Intelligent- CAT-Lab/RE2Bench/blob/main/results/input_prediction/gemini-3-pro-preview-reasoning/HC/sympy__sympy- 11822%40%40sympy.printing.conventions.py%40%40split_super_sub.txt

    2026

  57. [57]

    RE2-Bench. 2026. False Negative Detected from GPT-5-mini in Avatar. https://github.com/Intelligent-CAT-Lab/ RE2Bench/blob/main/results/input_prediction/gpt-5-mini-reasoning/LC/codeforces_379_A.txt

    2026

  58. [58]

    RE2-Bench. 2026. False Negative Detected from GPT-5-mini in Real-world Projects. https://github.com/Intelligent- CAT-Lab/RE2Bench/blob/main/results/input_prediction/gpt-5-mini-reasoning/HC/sympy__sympy-11822%40% 40sympy.printing.conventions.py%40%40split_super_sub.txt

    2026

  59. [59]

    Monoshi Kumar Roy, Simin Chen, Benjamin Steenhoek, Jinjun Peng, Gail Kaiser, Baishakhi Ray, and Wei Le. 2025. CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning.arXiv preprint arXiv:2506.00750(2025)

    work page arXiv 2025

  60. [60]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray

  61. [61]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm.Proceedings of the ACM on Software Engineering1, FSE (2024), 951–971

    2024

  62. [62]

    Ketan Rajshekhar Shahapure and Charles Nicholas. 2020. Cluster quality analysis using silhouette score. In2020 IEEE 7th international conference on data science and advanced analytics (DSAA). IEEE, 747–748

    2020

  63. [63]

    Aleksei Shestov, Rodion Levichev, Ravil Mussabayev, Evgeny Maslov, Pavel Zadorozhny, Anton Cheshkov, Rustam Mussabayev, Alymzhan Toleu, Gulmira Tolegen, and Alexander Krassovitskiy. 2025. Finetuning large language models for vulnerability detection.IEEE Access(2025)

    2025

  64. [64]

    Simeng Sun, Cheng-Ping Hsieh, Faisal Ladhak, Erik Arakelyan, Santiago Akle Serano, and Boris Ginsburg. 2025. L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution.arXiv preprint arXiv:2503.22832(2025)

    work page arXiv 2025

  65. [65]

    Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, Ke Wang, et al. 2025. EquiBench: Benchmarking Large Language Models’ Understanding of Program Semantics via Equivalence Checking.arXiv preprint arXiv:2502.12466(2025)

    work page arXiv 2025

  66. [66]

    Sean Williams and James Huckle. 2024. Easy problems that llms get wrong.arXiv preprint arXiv:2405.19616(2024)

    work page arXiv 2024

  67. [67]

    Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang, Chengpeng Wang, Lin Tan, and Xiangyu Zhang. 2025. CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks.arXiv preprint arXiv:2507.05269 (2025)

    work page arXiv 2025

  68. [68]

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

    2024

  69. [69]

    Rem Yang, Julian Dai, Nikos Vasilakis, and Martin Rinard. 2025. Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning.arXiv preprint arXiv:2504.05518(2025)

    work page arXiv 2025

  70. [70]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. InInternational Conference on Machine Learning. PMLR, 39818–39833

    2023

  71. [71]

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. 2024. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–12. , Vol. 1, No. 1, Article . Publication date: February 20...

    2024

  72. [72]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and improving chatgpt for unit test generation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1703–1726. , Vol. 1, No. 1, Article . Publication date: February 2018

    2024