Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Understanding Benchmark Language Under Weakened Formal Semantics

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Turning benchmark language into runnable Python programs lifts average accuracy by 32.7 percent, the paper claims.

desk verdict A well-built empirical systems paper whose central accuracy claim is confounded by transductive label leakage from the in-distribution retrieval pool; the computables analysis is the real contribution. read the letter →

arxiv 2509.17455 v2 pith:5PXPYO66 submitted 2025-09-22 cs.CL cs.AI

Haoyang Chen , Kumiko Tanaka-Ishii This is my paper · ORCID
classification cs.CLcs.AI
keywords text-to-codecode-as-reasoningretrieval-augmentedgenerationprogramsynthesiscodificationcomputablesiterativerefinementLLMreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how much of benchmark language an LLM can genuinely understand when full formal semantics are out of reach. Its answer is ICRAG, a system that compiles each natural-language task into a Python program, retrieves relevant rules and code snippets, executes the program, and iteratively repairs it until it runs and returns an answer. Across 13 benchmarks in math, causal reasoning, law, and medicine, ICRAG reports average accuracy 32.7 percentage points above text-only baselines, with the largest gains on the hardest sets. A sympathetic reader would take the contribution to be evidence that operational computables, programs whose execution is the reasoning, offer a practical, inspectable middle ground between formal proof semantics and black-box text.

What carries the argument

The load-bearing mechanism is the retrieve-execute-repair loop over a computable: an executable Python program induced from the task text, whose runtime behavior is taken as operational evidence that the program understands the language. ICRAG starts from an initial code draft, retrieves from a knowledge pool R1 (domain documents) and a k-fold construction R2 (code snippets generated from other instances of the same benchmark, with the current instance held out), executes the draft, uses exceptions and trace failures as repair signals, and asks the LLM for a refined program and a new retrieval query; convergence is declared when no query is produced, typically within three iterations for 96% of test queries.

What would settle it

Remove the R2 pool entirely, or swap it for snippets generated from a different benchmark with answers scrubbed, and rerun ICRAG on CAIL and PubMedQA; if accuracy falls to the R1-only or CoC level, the k-fold retrieval pool is the source of the gains, while if accuracy holds, the repair loop is doing the work. A second check is to split R2 by nearest-neighbor similarity and test whether accuracy is systematically higher on questions whose retrieved snippets share answer-encoding templates.

Watch

Extended reading notes

Core claim

The central claim is that weakening formal semantics down to executable behavior does not lose the ability to solve hard natural-language tasks: ICRAG's iterative retrieval-and-repair loop produces programs that outperform one-shot code execution and text-only chain-of-thought on every one of the 13 evaluated benchmarks, and outperform iterative text RAG on all of them as well. The program is not a side effect of the answer; it is the reasoning. Each iteration executes the current candidate, lets runtime failures expose missing conditions and exceptions, retrieves the minimal domain text or code fragment that fills the gap, and rewrites the program; the process stops when the model emits no further query. The paper presents this as showing that codification, previously limited to math-like questions, extends to causal, legal, and biomedical reasoning.

Load-bearing premise

The largest claimed gains rest on the assumption that code snippets drawn from the other four-fifths of the same benchmark, each containing its gold answer, do not let the model recover the held-out answer by analogy; if that assumption fails, the reported improvements could be answer leakage rather than retrieval-and-repair understanding.

Editorial extensions

If this is right

  • If ICRAG's results hold, law and medicine benchmarks become treatable as programming tasks, and their answers come with runnable artifacts an auditor can read and test.
  • Execution feedback, not just retrieval, is what pushes accuracy above iterative text RAG: the paper's IRCoT baseline also retrieves iteratively but never compiles to code, and ICRAG beats it on every dataset.
  • The bounded iteration count, N≤3 for 96% of queries, means the transparency gain does not require unbounded compute per question.
  • For algorithmic tasks the generated programs generalize: re-running them on modified valid inputs preserved or exceeded original accuracy, so the code has learned the rule rather than only the instance.
  • Harder domains produce measurably more complex computables, including deeper stacks and higher cyclomatic complexity, giving a code-based lens for comparing tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test would remove R2 entirely or use snippets from a disjoint corpus: the paper's own GitHub-only experiment suggests generic code alone yields smaller gains, implying that part of ICRAG's edge comes from in-domain analogies rather than external knowledge per se.
  • The k=5 construction means improvements on benchmarks like CAIL could partly reflect the model recognizing answer patterns from similar solved instances; a follow-up could vary k and measure accuracy on instances whose nearest R2 neighbors share the gold answer.
  • Treating execution failure as a semantic signal suggests a cheap consistency probe: mutate an input and check whether the revised program still encodes the same rule, which the paper's correctness metric only samples for 50 programs per dataset.
  • If computables become a standard artifact, benchmark quality itself could be reassessed by code-complexity statistics, for instance by flagging tasks whose generated programs are trivial conditionals as less informative.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ICRAG, a system that converts natural-language benchmark instances into Python programs via an LLM, augments each generation with retrieval from two sources (domain corpora R1 and a dataset-derived pool R2), and iteratively refines the program by executing it and feeding execution errors and retrieved snippets back to the LLM. The authors evaluate ICRAG on 13 benchmarks spanning math, causal reasoning, legal judgment, biomedical QA, and Big-Bench-Hard/Extra-Hard, reporting an average 32.7% absolute accuracy improvement over text-only and one-shot code-execution baselines. They also introduce the notion of "computables" and analyze the generated code's complexity, AST structure, and t-SNE embedding to argue that natural-language tasks yield a distinctive class of programs.

Significance. If the central accuracy claim held up, this would be a useful empirical contribution: it would show that iterative retrieval-augmented code generation with execution feedback generalizes codification from math puzzles to legal, medical, and causal-reasoning tasks, and that the resulting programs provide inspectable evidence of reasoning. The paper has notable strengths: it covers a broad benchmark suite, reports controlled external-knowledge experiments (E1-E3) that separately vary in-domain pool size and GitHub-code injection, provides the exact prompt templates and FAISS configuration in appendices, and includes cross-model robustness checks with Claude, Gemini, DeepSeek, GPT-3.5, and Llama. The AST-profile and complexity analyses of generated code are novel and could be of independent interest. However, the headline comparison against no-retrieval baselines is undermined by the transductive construction of R2, and the "correctness" metric for nongeneralizable tasks is a tautology as stated.

major comments (3)
  1. [§3.2, §4.1, Table 1] The retrieval pool R2 is built from a (k-1)/k portion of each benchmark itself, and each snippet s = LLM(d) is generated via Prompt A.1, which explicitly includes the gold answer and asks for a program that "returns the final answer." The k=5 split only excludes the exact test instance; it does not prevent transductive label leakage. For the classification-style datasets (CAIL with 150 classes, ECHR, CAP, PubMedQA, Health-Claim, BioASQ, and many BBH/BBEH tasks), the top-3 retrieved snippets are labeled in-distribution analogs, so the LLM can infer the held-out answer by nearest-neighbor analogy or by matching the retrieved code's printed label. Direct, CoT, and CoC see no such labeled examples, so the reported +32.7% average gain and the claim that ICRAG "consistently exceeds text-only reasoning and one-shot code execution" may reflect an unfair advantage rather than the method itself. The ablations RAG_NL, RAG_Code, and IRCoT share R2 and therefore do not control for this. Please add an ablation with R1 only, or rebuild R2 from a proper training split, or otherwise demonstrate that the gains over no-retrieval baselines persist when the retrieval pool contains no in-distribution labeled answers.
  2. [§5.3.3, Table 4] The paper states that for nongeneralizable tasks "the correctness is by definition the same as its one-time execution success on that instance" and Table 4 reports correctness values that are merely the original execution accuracies with small unexplained deviations. This is not a test of generalization; it is a restatement of the accuracy numbers. The claim in Section 5.3 that correctness "evaluates whether a generated program consistently produces correct outputs when applied to new valid inputs" is therefore not supported for the legal and health-claim datasets. Either remove the correctness framing for nongeneralizable tasks, or design a genuinely out-of-sample evaluation (e.g., human-validated input perturbations that change case facts while preserving the applicable legal or medical rule) and report how the perturbations were generated and verified.
  3. [§4.1, §4.2, Table 3] The ProofNet evaluation is inconsistent and under-validated. Section 4.1 says the OpenAI evaluation platform applies "the GPT-o3 model" to grade proofs, while Section 4.2 says "a GPT-3-based grader" was used. No evidence is provided that this automatic grader is reliable: there is no human validation sample, no agreement statistic, and no comparison with the benchmark's official protocol. Because ProofNet is one of the 13 datasets and shows one of the largest ICRAG improvements (0.74 to 0.98), the grader's reliability is load-bearing for that row. Please clarify which model was actually used, and provide validation of the grader on a held-out sample (e.g., human-judged correctness or agreement with a formal proof checker).
minor comments (5)
  1. [Table 3] Direct, CoT, and CoC columns report point estimates without variance, while the RAG-based methods include standard deviations; please state the number of runs/seeds for all methods and report variances uniformly.
  2. [Section 4.2] The phrase "GPT-3-based grader" in Section 4.2 contradicts the "GPT-o3" statement in Section 4.1; this should be reconciled and the grader version stated precisely.
  3. [Appendix E] The appendix states that the illustrative examples "are not actual dataset samples," but several examples are described as reflecting the dataset styles; please mark clearly which examples are synthetic and which, if any, are real dataset instances, and provide the exact prompts used for non-math domains (Appendix A says the templates are for "all Math experiments" but the method is applied to legal and medical tasks).
  4. [Figure 3] The unified panel for E1/E2/E3 is difficult to read: the x-axis is shared across two different meanings of r, and the E3 points at x=75% are confusing since E3 uses only GitHub code; please split the plots or clarify the axes and point labels.
  5. [Figure 5] The t-SNE embedding lacks methodological details such as perplexity, number of iterations, and distance metric; the informal axis labels "More Textual" and "More Complex Reasoning" should be justified or replaced with concrete feature directions.

Circularity Check

2 steps flagged · score 6.0 of 10

R2 retrieval pool leaks benchmark labels: ICRAG's headline accuracy gains are partly a transductive artifact of retrieving answer-encoding snippets from the same test distribution.

  1. fitted input called prediction [Section 3.2 (R2 construction), Section 4.1 (k=5), Table 3 note; Appendix A Prompts A.1 and A.2(ii)]
    "R2, on the other hand, is a collection of similar solved examples drawn from the dataset itself. ... we transform d∈(k−1)/k, including answers, from Dataset into a Python program snippet, i.e., s=LLM(d), via Prompt A.1 ... and we add s to R2. This ensures that the specific task being solved is held out from the retrieval pool ... ICRAG (final) uses the full retrieval pool (R1 + R2 = 100%)."

    Prompt A.1 is defined as: "Given a yes/no or free-form question and its correct answer, write an executable Python program that contains your reasoning in code and returns the final answer." Every R2 snippet therefore embeds the gold label of a benchmark instance, and Prompt A.2(ii) feeds "Similar Code Snippets: {context_code}" into the refiner. For a test item, the top-3 retrieved snippets are labeled analogs from the other 4/5 of the same dataset, so the model can transfer the label or decision logic by nearest-neighbor analogy. The k-fold split prevents exact instance lookup, not analogical label leakage.

  2. self definitional [Section 5.3.3, Table 4 note]
    "for these nongeneralizable tasks, our evaluation of the code is limited to the original instance, and the program's correctness is by definition the same as its one-time execution success on that instance."

    The correctness metric is introduced as "a stronger metric" that tests whether a program produces correct outputs on new valid inputs, but for nongeneralizable tasks the paper explicitly defines correctness to equal the one-shot execution accuracy already reported. Consequently, Table 4's nongeneralizable rows provide no independent evidence beyond the original accuracy measure. The limitation is acknowledged in the text, making this a secondary definitional collapse rather than a load-bearing step for the main method comparison.

full rationale

This is an empirical systems paper with no equation-level derivation chain, no imported uniqueness theorem, and no load-bearing self-citation; most of the method content (code-as-reasoning, iterative refinement, external-knowledge ablations, AST-based analysis) is self-contained. However, the central performance claim is partially circular by construction. R2 is built from 4/5 of each benchmark after converting each instance's question and correct answer into a Python snippet via Prompt A.1, and ICRAG's refiner receives those retrieved snippets as context. The k-fold split only excludes the exact test instance; it does not prevent the model from recovering the held-out answer through analogy to labeled same-distribution snippets. Because Direct, CoT, and CoC baselines see no such labeled in-distribution examples, the headline "+32.7%" and "consistently exceeds text-only reasoning" are inflated by transductive label leakage, not solely by iterative codification. The paper's own ablations (RAG_NL, RAG_Code, IRCoT) also use R2, so they share the leakage and do not isolate it. The correctness metric for nongeneralizable tasks is explicitly defined to collapse to the accuracy being evaluated, which the paper admits. These issues warrant a score of 6: the central evaluation is partially circular, while the method and code analyses retain independent content.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim is empirical and depends on several domain assumptions about benchmark quality, retrieval relevance, and the faithfulness of LLM-generated programs. The most fragile assumption is that the in-distribution retrieval pool does not leak answer patterns, since the pool is drawn from the same benchmarks being tested. Hyperparameters such as k, top_k, temperature, embedding model, and the correctness sample size are chosen by hand and are not justified beyond intuition.

free parameters (5)
  • k (number of folds for R2) = 5
    Chosen by hand to construct the retrieval pool; affects the amount of in-distribution examples available for retrieval and the size of the test split.
  • top_k retrieved snippets = 3
    Number of nearest-neighbor code snippets retrieved per query; set by hand in the retrieval-index configuration.
  • sampling temperature tau_code = 0.2
    Sampling temperature used for the CoC code generation; chosen by hand in Appendix B.1.
  • embedding model = all-MiniLM-L6-v2 (384 dim)
    Used for embedding passages and snippets in the FAISS index; chosen by hand.
  • number of programs sampled for correctness evaluation = 50 per dataset
    Small sample size used in the correctness evaluation for generalizable tasks; chosen by hand without justification.
assumptions (4)
  • domain assumption Benchmark ground-truth answers are correct.
    The accuracy metric assumes the labels in all 13 benchmarks are accurate; this is standard but not verified in the paper.
  • domain assumption LLM-generated code and retrieved snippets provide faithful evidence.
    The method relies on GPT-4o-Mini to produce meaningful programs and on FAISS retrieval to return relevant snippets; failures or hallucinations in these components would change results.
  • ad hoc to paper k-fold hold-out prevents answer leakage from R2.
    The paper assumes that excluding the target instance from the retrieval pool is sufficient to avoid leakage, even though the retrieved snippets encode answers to similar instances from the same benchmark distribution.
  • domain assumption External knowledge bases (R1) are appropriate for each task.
    Each task uses a designated corpus (e.g., U.S. Code for CAP, textbooks for USMLE); if these sources are incomplete or not aligned with the questions, the method's gains could be overstated.
invented entities (1)
  • computables independent evidence
    purpose: Executable Python programs generated from natural language task descriptions; they serve as operational evidence of semantic adequacy and as objects of analysis.
    The programs are concrete artifacts shown in the paper and could in principle be inspected and executed by others, although no dataset of computables is released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Benchmark Language Under Weakened Formal Semantics." pith.science (2026). https://pith.science/paper/5PXPYO66

@misc{pith2026250917455,
  author       = {Pith},
  title        = {Pith review of: Understanding Benchmark Language Under Weakened Formal Semantics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5PXPYO66}},
  note         = {Machine review of arXiv:2509.17455}
}
read the original abstract

State-of-the-art NLP benchmarks require interpretation of natural language that specifies conditions, procedures, and exceptions, often relying on implicit assumptions and external knowledge. Constructing complete semantic representations with proof-theoretic guarantees is frequently impractical at scale, and purely text-based reasoning offers limited means of inspection. This paper asks how much understanding of benchmark language can be achieved when formal semantic guarantees are weakened. We investigate this question by extracting computables: executable representations whose runtime behavior provides operational evidence of semantic adequacy, including executability, execution traces, and runtime failures. We induce and iteratively refine computables for benchmark instances using retrieval from external knowledge. Across mathematical reasoning, multi-step reasoning, causal inference, and rule- and exception-heavy legal and biomedical benchmarks, we find that the proposed approach consistently exceeds text-only reasoning and one-shot code execution. Beyond accuracy, our analyses show that these computables provide scalable, inspectable semantic evidence: they expose conditions and exceptions benchmark language forces into executable form, offering a practical bridge between proof-oriented semantics and purely textual reasoning.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 24 canonical work pages

  1. [1]

    https://arxiv.org/abs/2410.05229

    Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., Farajtabar, M.: GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024). https://arxiv.org/abs/2410.05229

    arXiv 2024

  2. [2]

    (eds.) Advances in Neural Information Processing Systems, vol

    Chi, H., Li, H., Yang, W., Liu, F., Lan, L., Ren, X., Liu, T., Han, B.: Unveiling causal reasoning in large Language Models: Reality or mirage? In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tom- czak, J., Zhang, C. (eds.) Advances in Neural Information Processing Systems, vol. 37, pp. 96640–96670. Curran Associates, Inc., ??? (2024). h...

    work page 2024

  3. [3]

    https://arxiv.org/abs/2410

    Yamin, K., Gupta, S., Ghosal, G.R., Lipton, Z.C., Wilder, B.: Failure Modes of LLMs for Causal Reasoning on Narratives (2024). https://arxiv.org/abs/2410. 23884 41

    work page 2024

  4. [4]

    In: Proceedings of the 36th International Conference on Neural Informa- tion Processing Systems

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in Large Language Models. In: Proceedings of the 36th International Conference on Neural Informa- tion Processing Systems. NIPS ’22. Curran Associates Inc., Red Hook, NY, USA (2024)

    work page 2024

  5. [5]

    In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F

    Li, C., Liang, J., Zeng, A., Chen, X., Hausman, K., Sadigh, D., Levine, S., Fei-Fei, L., Xia, F., Ichter, B.: Chain of Code: Reasoning with a language model-augmented code emulator. In: Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., Berkenkamp, F. (eds.) Proceedings of the 41st International Conference on Machine Learning...

    work page 2024

  6. [6]

    https://arxiv.org/abs/2211.12588

    Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2023). https://arxiv.org/abs/2211.12588

    arXiv 2023

  7. [7]

    In: Proceedings of the 40th International Conference on Machine Learning

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., Neubig, G.: PAL: program-aided Language Models. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23. JMLR.org, ??? (2023)

    work page 2023

  8. [8]

    In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval- augmented generation for knowledge-intensive NLP tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Infor- mation Processing Systems, vol. 33, pp. ...

    work page 2020

Show all 44 references
  1. [9]

    Science378(6624), 1092–1097 (2022) https://doi.org/10.1126/science.abq1158

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., Masson d’Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D.J., Su...

    2022 doi

  2. [10]

    https://arxiv.org/abs/2310.06770

    Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K.: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2024). https://arxiv.org/abs/2310.06770

    2024 arXiv

  3. [11]

    ACM Trans

    Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., Jiao, W.: Self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol.33(7) (2024) https://doi.org/10.1145/3672456

    2024 doi

  4. [12]

    In: Thirty-seventh Conference on Neural Information Processing Systems (2023)

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T.L., Cao, Y., Narasimhan, K.R.: Tree of thoughts: Deliberate problem solving with large language models. In: Thirty-seventh Conference on Neural Information Processing Systems (2023). https://openreview.net/forum?id=5Xc1ecxO1h 42

    2023

  5. [13]

    From Frege to Gödel: A Source Book in Mathematical Logic1931, 1–82 (1879)

    Frege, G.: Begriffsschrift, a formula language, modeled upon that of arithmetic, for pure thought [1879]. From Frege to Gödel: A Source Book in Mathematical Logic1931, 1–82 (1879)

  6. [14]

    Linguistics and Philosophy4(2), 159–219 (1981) https://doi.org/10.1007/bf00350139

    Barwise, J., Cooper, R.: Generalized quantifiers and natural language. Linguistics and Philosophy4(2), 159–219 (1981) https://doi.org/10.1007/bf00350139

    1981 doi

  7. [15]

    Transactions of the Association for Computational Linguistics 10, 1266–1284 (2022) https://doi.org/10.1162/tacl_a_00518

    Yanaka, H., Mineshima, K.: Compositional evaluation on japanese textual entail- ment and similarity. Transactions of the Association for Computational Linguistics 10, 1266–1284 (2022) https://doi.org/10.1162/tacl_a_00518

    2022 doi

  8. [16]

    Edin- burgh Advanced Textbooks in Linguistics, ??? (2016)

    Winter, Y.: Elements of Formal Semantics. Edin- burgh Advanced Textbooks in Linguistics, ??? (2016). https://books.google.com/books/about/Elements_of_Formal_Semantics.html?hl ¯&id¯gioONQEACAAJ

    2016

  9. [17]

    Computer Law & Security Review46, 105696 (2022) https://doi.org/ 10.1016/j.clsr.2022.105696

    Fungwacharakorn, W., Satoh, K.: Toward a practical legal rule revision in legal debugging. Computer Law & Security Review46, 105696 (2022) https://doi.org/ 10.1016/j.clsr.2022.105696

    2022

  10. [18]

    In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y

    Yasunaga, M., Ren, H., Bosselut, A., Liang, P., Leskovec, J.: QA-GNN: Rea- soning with Language Models and Knowledge Graphs for question answering. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zho...

    2021

  11. [19]

    https://arxiv.org/abs/2307.07697

    Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.-Y., Guo, J.: Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2024). https://arxiv.org/abs/2307.07697

    2024 arXiv

  12. [20]

    In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.-t

    Parvez, M.R., Ahmad, W., Chakraborty, S., Ray, B., Chang, K.-W.: Retrieval augmented code generation and summarization. In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.-t. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2719–2734. Associatio...

    2021 doi

  13. [21]

    (eds.) Findings of the Association for Computational Linguistics: NAACL 2025, pp

    Wang, Z.Z., Asai, A., Yu, X.V., Xu, F.F., Xie, Y., Neubig, G., Fried, D.: CodeRAG-bench: Can retrieval augment code generation? In: Chiruzzo, L., Ritter, A., Wang, L. (eds.) Findings of the Association for Computational Linguistics: NAACL 2025, pp. 3199–3214. Association for C...

    2025 doi

  14. [22]

    https: //arxiv.org/abs/2411.04444

    Liu, B., Jiang, Y., Zhang, Y., Niu, N., Li, G., Liu, H.: An Empirical Study on the Potential of LLMs in Automated Software Refactoring (2024). https: //arxiv.org/abs/2411.04444

    2024 arXiv

  15. [23]

    In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering

    Pomian, D., Bellur, A., Dilhara, M., Kurbatova, Z., Bogomolov, E., Sokolov, A., Bryksin, T., Dig, D.: Em-assist: Safe automated extractmethod refactoring with llms. In: Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. F...

    2024

  16. [24]

    Automated Software Engg.32(1) (2025) https://doi.org/10.1007/s10515-024-00485-2

    Qu, Y., Huang, S., Li, Y., Bai, T., Chen, X., Wang, X., Li, L., Yao, Y.: Badcodeprompt: backdoor attacks against prompt engineering of large lan- guage models for code generation. Automated Software Engg.32(1) (2025) https://doi.org/10.1007/s10515-024-00485-2

    2025 doi

  17. [25]

    https://arxiv.org/abs/2212.10509

    Trivedi, H., Balasubramanian, N., Khot, T., Sabharwal, A.: Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2023). https://arxiv.org/abs/2212.10509

    2023 arXiv

  18. [26]

    https://arxiv.org/abs/2110.14168

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training Verifiers to Solve Math Word Problems (2021). https://arxiv.org/abs/2110.14168

    2021 arXiv

  19. [27]

    Azerbayev, Z., Piotrowski, B., Schoelkopf, H., Ayers, E.W., Radev, D., Avi- gad, J.: ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics (2023)

    2023

  20. [28]

    In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023)

    Yu, W., Jiang, M., Clark, P., Sabharwal, A.: IfQA: A dataset for open-domain question answering under counterfactual presuppositions. In: The 2023 Conference on Empirical Methods in Natural Language Processing (2023)

    2023

  21. [29]

    https: //arxiv.org/abs/2306.05836

    Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R., Diab, M., Schölkopf, B.: Can Large Language Models Infer Causation from Correlation? (2024). https: //arxiv.org/abs/2306.05836

    2024 arXiv

  22. [30]

    https: //arxiv.org/abs/2210.09261

    Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H.W., Chowdhery, A., Le, Q.V., Chi, E.H., Zhou, D., Wei, J.: Challenging BIG- Bench Tasks and Whether Chain-of-Thought Can Solve Them (2022). https: //arxiv.org/abs/2210.09261

    2022 arXiv

  23. [31]

    https://arxiv.org/abs/2502.19187

    Kazemi, M., Fatemi, B., Bansal, H., Palowitch, J., Anastasiou, C., Mehta, S.V., Jain, L.K., Aglietti, V., Jindal, D., Chen, P., Dikkala, N., Tyen, G., Liu, X., Shalit, U., Chiappa, S., Olszewska, K., Tay, Y., Tran, V.Q., Le, Q.V., Firat, O.: BIG-Bench Extra Hard (2025). https:...

    2025 arXiv

  24. [32]

    https://arxiv.org/abs/1807.02478

    Xiao, C., Zhong, H., Guo, Z., Tu, C., Liu, Z., Sun, M., Feng, Y., Han, X., Hu, 44 Z., Wang, H., Xu, J.: CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction (2018). https://arxiv.org/abs/1807.02478

    2018 arXiv

  25. [33]

    In: Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics

    Chalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., Malakasiotis, P.: Paragraph-level rationale extraction through regularization: A case study on European Court of Human Rights cases. In: Proceedings of the Annual Conference of the North Ameri...

    2021

  26. [34]

    https://arxiv.org/abs/2211.00582

    Semo, G., Bernsohn, D., Hagag, B., Hayat, G., Niklaus, J.: ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the US (2022). https://arxiv.org/abs/2211.00582

    2022 arXiv

  27. [35]

    Jin, Q., Dhingra, B., Liu, Z., Cohen, W., Lu, X.: Pubmedqa: A dataset for biomedical research question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processin...

    2019

  28. [36]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp

    Kotonya, N., Toni, F.: Explainable automated fact-checking for public health claims. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7740–7754. Association for Compu- tational Linguistics, Online (2020).https://www.aclweb...

    2020

  29. [37]

    García-Ferrero, I., Agerri, R., Salazar, A.A., Cabrio, E., Iglesia, I., Lavelli, A., Magnini, B., Molinet, B., Ramirez-Romero, J., Rigau, G., Villa-Gonzalez, J.M., Villata, S., Zaninello, A.: Medical mT5: An Open-Source Multilingual Text-to-Text LLM for The Medical Domain (2024)

    2024

  30. [38]

    https://github.com/google-research/google-research/tree/master/mbpp

    google-research/mbpp at master·google-research/google-research — github.com. https://github.com/google-research/google-research/tree/master/mbpp. [Accessed 12-05-2025]

    2025

  31. [39]

    https://huggingface.co/datasets/PatrickHaller/the-stack-python-1M

    PatrickHaller/the-stack-python-1M·Datasets at Hugging Face — huggingface.co. https://huggingface.co/datasets/PatrickHaller/the-stack-python-1M. [Accessed 12-05-2025]

    2025

  32. [40]

    https://huggingface.co/datasets/notbadai/python_functions_reasoning

    notbadai/python_functions_reasoning·Datasets at Hugging Face — hugging- face.co. https://huggingface.co/datasets/notbadai/python_functions_reasoning. [Accessed 12-05-2025]

    2025

  33. [41]

    https://arxiv.org/abs/2411.04905 45

    Huang, S., Cheng, T., Liu, J.K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L., Yuan, R., Zhang, Z., Fu, J., Liu, Q., Zhang, G., Wang, Z., Qi, Y., Xu, Y., Chu, W.: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models (2025). https://arxiv.or...

    2025 arXiv

  34. [42]

    MIT Press, ??? (2009)

    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Intro- duction to Algorithms, Third Edition. MIT Press, ??? (2009). https://books.google.com/books/about/Introduction_to_Algorithms_third_edition.html?hl ¯&id¯ı- bUBQAAQBAJ

    2009

  35. [43]

    IEEE Transactions on Software Engineering SE-2(4), 308–320 (1976) https://doi.org/10.1109/TSE.1976.233837

    McCabe, T.J.: A complexity measure. IEEE Transactions on Software Engineering SE-2(4), 308–320 (1976) https://doi.org/10.1109/TSE.1976.233837

    1976

  36. [44]

    https://docs.python.org/3/ library/ast.html

    ast — Abstract Syntax Trees — docs.python.org. https://docs.python.org/3/ library/ast.html. [Accessed 14-05-2025] 46

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.