Pith. sign in

REVIEW 4 major objections 6 minor 47 references

State-of-the-art LLMs can reason about complex Python execution paths well enough to generate covering test cases and flag division-by-zero bugs, though stronger reasoners do not always win.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:45 UTC pith:WNJ5GGVW

load-bearing objection A useful benchmark and an interesting overthinking finding, but the main RQ1 accuracy numbers are potentially inflated by training-data contamination and the RQ2 labels lack an independent oracle. the 4 major comments →

arxiv 2511.18288 v2 pith:WNJ5GGVW submitted 2025-11-23 cs.SE

Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python

Wenhan Wang , Kaibo Liu , Zeyu Sun , An Ran Chen , Ge Li , Gang Huang , Lei Ma This is my paper
classification cs.SE
keywords execution path reasoningsymbolic executionlarge language modelstest case generationpath classificationbug detectionPythontest coverage
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that large language models can serve as path constraint solvers for symbolic execution, especially in Python, where solver-based tools are impractical. It evaluates LLMs on two tasks: generating test inputs that satisfy a given execution path, and classifying paths as feasible, infeasible, or division-by-zero-triggering. Results show top models generate correct test cases for over 60% of complex competition-level paths, achieve up to 83% classification accuracy, and improve coverage in real-world repositories. Notably, stronger reasoning models do not always outperform weaker ones—reasoning models tend to 'overthink' in classification, hurting accuracy. A sympathetic reader would see this as evidence that LLM-based path reasoning is a viable complement to traditional symbolic execution.

Core claim

The paper claims that state-of-the-art LLMs can solve path constraints in both generation and classification settings: given a serialized execution path with branch conditions and loop iteration counts, LLMs can generate inputs that traverse that path (up to 65.6% path accuracy on competition-level programs), classify paths as valid, infeasible, or division-by-zero (up to 82.9% accuracy), and improve line coverage in real-world repositories where solver-based symbolic execution cannot run. At the same time, LRMs with extended chain-of-thought reasoning do not uniformly dominate: their overthinking degrades classification accuracy, and on real-world coverage, strong non-reasoning models match

What carries the argument

The central object is the 'execution path,' represented as a sequence of code blocks and branch conditions, with loop iterations explicitly annotated. The paper builds prompts that serialize this path (marking condition statements and loop iterations) and ask the LLM either to generate test inputs or to output a class label (VALID, INVALID, ZERODIVISION). Correctness for generation is measured by executing the generated test and comparing its trace against the ground-truth path; classification ground truth comes from manual annotation. This path-as-prompt mechanism is what carries all three research questions.

Load-bearing premise

The ground-truth labels for the 2,010 classification paths come from manual annotation without an executable check, so if any path is mislabeled as valid, invalid, or bug-triggering, the reported classification accuracies and bug-detection counts are directly wrong.

What would settle it

Run each of the 2,010 extracted paths through a Python interpreter with a concrete input search (concolic or random) to independently verify the manual labels; a path labeled invalid that executes to completion, or a path labeled valid that no input can execute, would disprove the classification claims.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • LLM-based path constraint solving could provide test case generation and bug detection for Python, a language lacking mature symbolic execution tools.
  • Path classification could help prune infeasible paths during symbolic execution, mitigating the path explosion problem.
  • The observed 'overthinking' failure of LRMs suggests that controlling reasoning effort or early-stopping is important for classification tasks.
  • Coding-specialized LLMs do not necessarily excel at execution-path reasoning, indicating a distinct capability worth training for.
  • Since path reasoning results are verifiable by execution, reinforcement learning from verifiable rewards could strengthen smaller models on this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the results transfer, LLM-based path reasoning could become a general fallback for languages where symbolic execution infrastructure is weak, not just Python.
  • A natural extension is to combine LLMs with SMT solvers: use LLMs to handle external API calls or data-structure-heavy constraints, and let solvers verify or refine the results.
  • The manual-label dependence suggests that future benchmarks should include executable oracles or concolic checks to make classification accuracies trustworthy.
  • The overthinking finding hints that task-specific reasoning budgets—shorter chain-of-thought for classification, longer for generation—could improve both accuracy and cost.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical study of whether LLMs can solve execution path constraints in Python. It constructs three evaluations: (RQ1) generating test inputs that exactly reproduce a given execution trace for LeetCode/TestEval programs; (RQ2) classifying CFG-extracted paths as valid, invalid, or zero-division-triggering, with labels obtained from a Google runtime-error dataset; and (RQ3) generating tests for partially covered functions from real-world repositories using execution-path prompts. The authors report that state-of-the-art models, especially reasoning models, achieve >65% path accuracy in RQ1, up to 82.9% classification accuracy in RQ2, and small coverage improvements in RQ3, and conclude that LLMs can complement or potentially replace symbolic execution for Python path constraint solving.

Significance. If the results hold, the paper would provide a useful evaluation framework and evidence that LLM-based path constraint solving is feasible for Python, where SMT-based tools are weak. The paper has concrete strengths: an executable evaluation pipeline for generated test cases (traces are compared by execution), a released benchmark and scripts, a broad model set (14 models), and a manually derived error taxonomy for reasoning chains. However, the central claims currently rest on two unverified ground-truth constructions — LeetCode example paths for RQ1 and manual annotations for RQ2 — and on small, unquantified coverage gains in RQ3. These issues are load-bearing and need to be addressed before the findings can be relied upon.

major comments (4)
  1. [§3.1.1, Table 1, Fig. 2(a)] The RQ1 ground-truth paths are obtained by running the 2–3 example test cases from each LeetCode problem statement, and the prompt shows the function name and the executed code. Because TestEval is public and LeetCode examples are widely distributed in pretraining corpora, the reported >60% path accuracies may partly reflect memorization or retrieval of example inputs rather than actual path-constraint reasoning. No contamination control is reported (e.g., held-out non-example inputs, newly written problems, or an analysis of whether outputs reproduce example inputs verbatim). This is load-bearing for Finding 1 and for the abstract's central claim; please add such a control or substantially soften the claim.
  2. [§3.2.1, Table 3] The RQ2 ground-truth labels are manually annotated with no executable oracle and no reported inter-annotator agreement. A path marked INVALID may be satisfiable under a different loop-unrolling bound (max_loop_iterations=2) or with a different input, and a VALID path is not confirmed by execution. The manual construction and appending of missing paths in §3.2.1 further risks selection bias. Since every accuracy, F1 score, and bug-detection count in Table 3 depends on these labels, the classification results lack independent verification. Provide an executable feasibility oracle where possible, report inter-annotator agreement, and/or release the annotations for independent audit.
  3. [Table 3, §4.2] The path classification dataset is highly imbalanced: 1,445 of 2,010 paths (71.9%) are ZeroDivision, so a trivial classifier that always predicts ZeroDivision achieves 71.9% accuracy. Many evaluated models are below this baseline (e.g., GPT-4.1 61.3%, o4-mini 43.2%, DeepSeek-R1 36.2%), and only Qwen3-235B exceeds it substantially. Reporting raw accuracy therefore overstates RQ2 performance. Report balanced accuracy or macro-F1, and compare against a majority-class baseline. The claim of 'up to 83% accuracy' should be framed against that baseline.
  4. [Table 5, Finding 5] The RQ3 coverage improvements over the gold tests are small in absolute terms (overall improvements of 0.2–4.8 percentage points), and the additional gain of path prompting over the baseline is at most +1.1 percentage points overall, with Qwen3-thinking showing a negative overall gain. No significance tests, confidence intervals, or effect sizes are provided. The statement that providing execution paths 'consistently improves' line coverage is stronger than the table supports. Add a statistical comparison of Path versus Baseline across functions/models, or restrict the claim to the cov@Pass measure where the effect is more visible.
minor comments (6)
  1. [Title] The manuscript title ('Can Large Language Models Solve Path Constraints in Symbolic Execution?') differs from the arXiv title ('Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python'). Please ensure consistency in the final version.
  2. [§4.1] The sentence 'The final human evaluation results are shown in Figure 2' should refer to Table 2, not Figure 2.
  3. [Algorithm 1] Line 9 uses 'match for i in range(x):' as pseudo-code, and the helper 'eval_loop' is not defined. This makes the traversal algorithm hard to follow; please replace with precise loop-handling conditions.
  4. [§3.3.1] TestGenEvallite is introduced without a description of its relationship to TestGenEval or the filtering criteria. Clarify how the 82 files and 605 functions were selected and whether the subset is representative.
  5. [§3.4.1] GPT-oss is categorized as a reasoning model (LRM), but the paper does not justify this classification. Please provide evidence or a citation for its reasoning characteristics, or move it to the non-reasoning group.
  6. [Front matter / Table 6] The front matter still contains template placeholders ('Do Not Use This Code', ACM Reference Format with 2018 dates). Table 6's column headers for Gemma3 and Qwen3 combine multiple model sizes ambiguously; split them to match Table 1.

Circularity Check

0 steps flagged

No significant circularity: the reported accuracies and coverage gains are direct empirical measurements against executable or manually assigned external oracles.

full rationale

The paper's central results are direct empirical measurements, not derived predictions. RQ1 builds ground-truth paths by running example LeetCode test cases through a modified trace module (§3.1.1), then executes the LLM-generated test case and compares its traced path to the ground truth (§3.1.2). Path accuracy is therefore an executable check, not an output of any fitted parameter. RQ2 constructs paths via bounded CFG traversal and manually annotates them as valid/invalid/ZeroDivision (§3.2.1); the LLM outputs are compared against those human-assigned labels. Even if manual labels are imperfect, that is a data-quality or validity threat, not definitional circularity. RQ3 measures line coverage with pytest for tests generated with and without execution-path prompts (§3.3.2). No quantity is defined in terms of the LLM outputs, and no parameter is fit to the evaluated data. The self-citations to TestEval [38] and LLM-Sym [37] are used as a data source and related-work context; the benchmark paths are independently recomputed by running example cases with a modified trace module, and the central measurements do not reduce to these prior works. Concerns about LeetCode-example memorization or manual-label accuracy are contamination/external-validity risks, not circularity. The paper's own §6 limitations (no advanced prompting, no direct comparison with traditional symbolic execution) are acknowledged scope limitations rather than circular steps. No specific reduction of any result to its inputs could be exhibited, so per the hard rules no circular step is reported.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The paper introduces no new entities; its claims rest on experimental design choices (loop limits, path caps) and domain assumptions about the correctness of trace/CFG extraction and manual labels. The free parameters listed are hand-set thresholds that shape the benchmark difficulty and therefore the reported accuracy numbers.

free parameters (5)
  • max_loop_iterations = 2
    CFG traversal unrolls loops at most twice (Algorithm 1, K=2). Feasible paths requiring more iterations are never generated, biasing the RQ2/RQ3 path sets toward short-loop paths.
  • max_path_length = 100 basic blocks
    CFG traversal stops paths longer than 100 blocks, excluding long paths from classification and real-world test generation.
  • max_paths = 50
    Traversal stops once 50 paths are collected per program, limiting coverage of the path space.
  • rq1_path_length_cap = 1000 statements
    Example-derived paths over 1,000 statements are discarded; this removes the longest, hardest paths from the RQ1 benchmark.
  • reasoning_effort = medium (o3/o4-mini)
    Proprietary LRM reasoning effort is fixed at medium; results may change with effort.
axioms (4)
  • domain assumption Branch outcomes in execution traces are correctly inferred by inspecting the next executed statement after a branch.
    Used to build ground-truth paths in §3.1.1; an incorrect inference would corrupt the path labels and the accuracy measurement.
  • domain assumption Scalpel CFG construction is faithful for the Python programs studied.
    RQ2/RQ3 paths are generated from Scalpel CFGs (§3.2.1); if CFG edges are wrong, extracted paths and their feasibility labels are wrong.
  • domain assumption Manual annotation of path classes (valid/invalid/zero-division) is correct.
    The RQ2 ground truth is assigned by the authors without an executable oracle or inter-annotator agreement; mislabeled paths directly change classification scores.
  • domain assumption Paths from example test cases in LeetCode descriptions represent the target path space.
    RQ1 ground truth is derived only from example tests; the benchmark does not sample from the full path space.

pith-pipeline@v1.3.0-alltime-deepseek · 21118 in / 11430 out tokens · 110002 ms · 2026-08-03T20:45:33.223122+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python." pith.science (2026). https://pith.science/paper/WNJ5GGVW

@misc{pith2026251118288,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Reason About Complex Execution Paths? An Empirical Study on Python},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNJ5GGVW}},
  note         = {Machine review of arXiv:2511.18288}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Execution path reasoning is a key step towards program semantics understanding. It is crucial for generating test cases that cover certain branches/paths, or detecting bugs that are triggered by some paths without actually executing the program. Traditionally, execution path reasoning can be achieved by symbolic execution techniques, but existing SMT-based symbolic execution approaches struggle with complex data structures and external API calls. This challenge is even more pronounced in languages with highly flexible syntax, such as Python, resulting in a lack of widely adopted tools for reasoning on execution paths. Therefore, reasoning execution paths with AI-based approaches become a promising direction. In this paper, we investigate the feasibility of adopting large language models (LLMs) for execution path reasoning on Python, where traditional path-based symbolic execution tools are unavailable. We conduct an empirical study on two types of path reasoning tasks: generation tasks for test case generation and classification tasks for bug detection. We build new evaluation pipelines and benchmarks from both competition-level programs and real-world repositories. Our results show that state-of-the-art LLMs can perform correct reasoning on execution paths and improve test coverage on real-world software, though models with stronger reasoning abilities do not always outperform weaker ones. These findings highlight the potential of utilizing LLMs as a complementary heuristic for path-aware code reasoning, especially in program languages lacking mature symbolic execution tools. We have released our benchmark and evaluation scripts at https://github.com/jacobwwh/llm-path-study.

Figures

Figures reproduced from arXiv: 2511.18288 by An Ran Chen, Gang Huang, Ge Li, Kaibo Liu, Lei Ma, Wenhan Wang, Zeyu Sun.

Figure 1
Figure 1. Figure 1: An overview of our empirical study [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The prompt templates for our tasks. (a): Test case generation (competition-level) with a short example [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The path accuracy across different execution path lengths (measured by the number of branch [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An example of the LRM DeepSeek-R1 successfully generates a correct test case for the given execution [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Heatmap of the distribution of the predicted execution path labels vs. the ground-truth labels. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: An example of the execution path (left) and reasoning CoT (right, generated by DeepSeek-R1) when [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 11 linked inside Pith

  1. [1]

    Jordi Armengol-Estapé, Quentin Carbonneaux, Tianjun Zhang, Aram H Markosyan, Volker Seeker, Chris Cummins, Melanie Kambadur, Michael FP O’Boyle, Sida Wang, Gabriel Synnaeve, et al. 2025. What I cannot execute, I do not understand: Training and Evaluating LLMs on Program Execution Traces.arXiv preprint arXiv:2503.05703(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    Roberto Baldoni, Emilio Coppa, Daniele Cono D’elia, Camil Demetrescu, and Irene Finocchi. 2018. A survey of symbolic execution techniques.ACM Computing Surveys (CSUR)51, 3 (2018), 1–39

    2018

  3. [3]

    David Bieber, Rishab Goel, Dan Zheng, Hugo Larochelle, and Daniel Tarlow. 2023. Static Prediction of Runtime Errors by Learning to Execute Programs with External Resource Descriptions. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=lLp-C5nTdJG

    2023

  4. [4]

    Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. InOSDI, Vol. 8. 209–224

    2008

  5. [5]

    Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2025. Reasoning Runtime Behavior of a Program with LLM: How Far are We?. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1869–1881

    2025

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  7. [7]

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024. Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Alberto Coen-Porisini, Giovanni Denaro, Carlo Ghezzi, and Mauro Pezzé. 2001. Using symbolic execution for verifying safety-critical systems. InProceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering. 142–151

    2001

  9. [9]

    Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An efficient SMT solver. InInternational conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337–340

    2008

  10. [10]

    Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. Semcoder: Training code language models with comprehensive semantics reasoning.Advances in Neural Information Processing Systems37 (2024), 60275–60308

    2024

  11. [11]

    PyExSMT: Python Symbolic Execution

    GitHub 2017. PyExSMT: Python Symbolic Execution. https://github.com/FedericoAureliano/PyExSMT

    2017

  12. [12]

    CrossHair

    GitHub 2025. CrossHair. https://github.com/pschanely/CrossHair

    2025

  13. [13]

    Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. InInternational Conference on Machine Learning. PMLR, 16568–16621

    2024

  14. [14]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025)

    Pith/arXiv arXiv 2025

  15. [15]

    Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, et al. 2025. Can large language models detect errors in long chain-of-thought reasoning? arXiv preprint arXiv:2502.19361(2025)

    Pith/arXiv arXiv 2025

  16. [16]

    Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. 2025. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. InThe Thirteenth International Conference on Learning Representations. https://openreview. net/forum?id=7o6SG5gVev

    2025

  17. [17]

    Zongze Jiang, Ming Wen, Jialun Cao, Xuanhua Shi, and Hai Jin. 2024. Towards understanding the effectiveness of large language models on directed test input generation. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1408–1420

    2024

  18. [18]

    Yeo Wei Jie, Ranjan Satapathy, Rick Goh, and Erik Cambria. 2024. How Interpretable are Reasoning Explanations from Prompting Large Language Models?. InFindings of the Association for Computational Linguistics: NAACL 2024. , Vol. 1, No. 1, Article . Publication date: November 2018. 20 Anonymous et al. 2148–2164

    2024

  19. [19]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=VTF8yNQM66

    2024

  20. [20]

    James C King. 1976. Symbolic execution and program testing.Commun. ACM19, 7 (1976), 385–394

    1976

  21. [21]

    Haonan Li, Yu Hao, Yizhuo Zhai, and Zhiyun Qian. 2024. Enhancing static analysis for practical bug detection: An llm-integrated approach.Proceedings of the ACM on Programming Languages8, OOPSLA1 (2024), 474–499

    2024

  22. [22]

    Li Li, Jiawei Wang, and Haowei Quan. 2022. Scalpel: The python static analysis framework.arXiv preprint arXiv:2202.11840(2022)

    Pith/arXiv arXiv 2022

  23. [23]

    Yihe Li, Ruijie Meng, and Gregory J Duck. 2025. Large Language Model powered Symbolic Execution.arXiv preprint arXiv:2505.13452(2025)

    arXiv 2025

  24. [24]

    Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. 2025. From system 1 to system 2: A survey of reasoning large language models.arXiv preprint arXiv:2502.17419(2025)

    Pith/arXiv arXiv 2025

  25. [25]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437(2024)

    Pith/arXiv arXiv 2024

  26. [26]

    Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. NExT: Teaching Large Language Models to Reason about Code Execution. InInternational Conference on Machine Learning. PMLR, 37929–37956

    2024

  27. [27]

    Introducing GPT-4.1 in the API

    OpenAI 2025. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/

    2025

  28. [28]

    Introducing gpt-oss

    OpenAI 2025. Introducing gpt-oss. https://openai.com/index/introducing-gpt-oss/

    2025

  29. [29]

    Introducing OpenAI o3 and o4-mini

    OpenAI 2025. Introducing OpenAI o3 and o4-mini. https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  30. [30]

    https://openrouter.ai/

    OpenRouter 2023. https://openrouter.ai/

    2023

  31. [31]

    Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021. CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)

    2021

  32. [32]

    https://github.com/pytest-dev/pytest

    pytest 2025. https://github.com/pytest-dev/pytest

    2025

  33. [33]

    2015.{Under-Constrained} symbolic execution: Correctness checking for real code

    David A Ramos and Dawson Engler. 2015.{Under-Constrained} symbolic execution: Correctness checking for real code. In24th USENIX Security Symposium (USENIX Security 15). 49–64

    2015

  34. [34]

    Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray

  35. [35]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report.arXiv preprint arXiv:2503.19786 (2025)

    Pith/arXiv arXiv 2025

  36. [36]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936

    2024

  37. [37]

    Wenhan Wang, Kaibo Liu, An Ran Chen, Ge Li, Zhi Jin, Gang Huang, and Lei Ma. 2024. Python symbolic execution with llm-powered code generation.arXiv preprint arXiv:2409.09271(2024)

    Pith/arXiv arXiv 2024

  38. [38]

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025. TESTEVAL: Benchmarking Large Language Models for Test Case Generation. InFindings of the Association for Computational Linguistics: NAACL 2025. 3547–3562

    2025

  39. [39]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  40. [40]

    Yaoxuan Wu, Xiaojie Zhou, Ahmad Humayun, Muhammad Ali Gulzar, and Miryung Kim. 2025. Generating and Understanding Tests via Path-Aware Symbolic Execution with LLMs.arXiv preprint arXiv:2506.19287(2025)

    arXiv 2025

  41. [41]

    Xiufeng Xu, Fuman Xie, Chenguang Zhu, Guangdong Bai, Sarfraz Khurshid, and Yi Li. 2025. Identifying Multi- parameter Constraint Errors in Python Data Science Library API Documentation.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1536–1558

    2025

  42. [42]

    Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang, Tingyu Lin, Weishan Zhao, Li Zhu, Hari Sundaram, et al. 2024. CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  43. [43]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025). , Vol. 1, No. 1, Article . Publication date: November 2018. Can Large Language Models Solve Path Constraints in Symbolic Execution? 21

    Pith/arXiv arXiv 2025

  44. [44]

    Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2024. Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning. InThe Twelfth International Conference on Learning Representa- tions. https://openreview.net/forum?id=6okaSfANzh

    2024

  45. [45]

    Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024. Chain of preference optimization: Improving chain-of-thought reasoning in llms.Advances in Neural Information Processing Systems37 (2024), 333–356

    2024

  46. [46]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5673–5684. Received 20 February 2007; revised 12 ...

    2023

  47. [2024]

    Code-aware prompting: A study of coverage-guided test generation in regression setting using llm.Proceedings of the ACM on Software Engineering1, FSE (2024), 951–971

    2024