Pith. sign in

REVIEW 4 major objections 38 references

LLMs catch falsified ISO software-engineering definitions yet reject many correct ones, revealing rejection bias rather than genuine understanding of SE terminology.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-08 20:19 UTC pith:GQTYMU3G

load-bearing objection Asymmetric pattern is real and useful; the leap to “rejection bias, not understanding” over-reads ISO wording as unique ground truth. the 4 major comments →

arxiv 2607.06004 v1 pith:GQTYMU3G submitted 2026-07-07 cs.SE

Large Language Models Have Unreliable Understanding of Software Engineering Terminology

Huzaifa Ejaz , Fabian C. Pe\~na , Steffen Herbold This is my paper
classification cs.SE
keywords large language modelssoftware engineering terminologyISO/IEC/IEEE 24765vocabulary understandingrejection biasdefinition classificationreasoning tokensSE standards
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether large language models truly understand standardized software-engineering terminology drawn from ISO/IEC/IEEE 24765:2017. The authors prompt models with both the authentic definitions and systematically falsified versions created by swapping key terms or stripping critical clauses, then score binary classification accuracy and the sensibility of any reasoning tokens the models produce. Most models detect the falsified definitions at high rates, yet they also reject a substantial fraction of the genuine definitions. The authors read this pattern as a systematic rejection bias rather than discriminative knowledge of the standard vocabulary. Explicit reasoning instructions do not reliably improve the results and can degrade them through over-thinking. If the finding holds, developers and agentic systems that rely on LLMs for terminology-sensitive SE work risk silent miscommunication even when the models appear confident.

Core claim

State-of-the-art LLMs detect semantically and structurally falsified definitions from the ISO/IEC/IEEE 24765:2017 Systems and Software Engineering Vocabulary with high accuracy, yet they also reject many of the authentic definitions. The pattern indicates a systematic rejection bias rather than genuine discriminative understanding of standardized SE terminology; explicit reasoning does not consistently improve discrimination and may hinder it.

What carries the argument

The controlled contrast between authentic ISO/IEC/IEEE 24765:2017 definitions and two families of systematic falsifications (semantic substitution of key terms; structural removal of critical information), evaluated by classification accuracy and by whether the model’s reasoning tokens correctly track the definitional content.

Load-bearing premise

The study treats the ISO definitions as unambiguous ground truth and treats success at binary classification of correct versus hand-crafted falsified definitions, plus sensible reasoning tokens, as a sufficient operationalization of “understanding” SE terminology.

What would settle it

Re-run the same correct-versus-semantic/structural-falsification protocol on a held-out sample of ISO 24765 definitions with an independent LLM family; if that family accepts authentic definitions at rates comparable to its detection of falsifications (balanced precision and recall near expert levels) and its reasoning tokens correctly isolate the critical distinguishing clauses, the rejection-bias claim is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLMs used on SE tasks that depend on precise terminology may silently reject valid standard definitions and produce inconsistent guidance.
  • Agentic or multi-step LLM pipelines that assume internal consistency of SE vocabulary inherit the same rejection bias.
  • Explicit chain-of-thought or reasoning prompts are not a reliable remedy and can introduce over-thinking that worsens terminology judgments.
  • Human developers acting on LLM-generated text that mishandles standard terms risk real-world miscommunication and defects.
  • Evaluations of LLMs for SE should include separate true-positive and true-negative rates on terminology probes rather than end-task accuracy alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same rejection bias may appear for other standardized SE vocabularies (IEEE standards, SWEBOK, etc.), pointing to a broader failure mode for domain-standard language.
  • Training or alignment objectives that reward cautious rejection of uncertain claims could produce the observed bias and might be mitigated by retrieval grounding against the standard itself.
  • Benchmarks that report only aggregate accuracy on terminology tasks will mask the imbalance; separate acceptance and rejection rates should become standard reporting practice.
  • A practical mitigation worth testing is to force the model to quote or retrieve the ISO definition before rendering a correctness judgment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The manuscript reports an empirical study of whether state-of-the-art LLMs understand standardized software-engineering terminology, operationalized as binary classification of ISO/IEC/IEEE 24765:2017 definitions presented either in their original form or after systematic semantic (key-term substitution) and structural (critical-information removal) falsification. The authors measure classification accuracy and the sensibility of generated reasoning tokens. They report that most models detect falsified definitions with high accuracy yet also reject many correct ISO definitions, and interpret this pattern as a systematic rejection bias rather than genuine discriminative understanding; explicit reasoning is found not to help consistently and may degrade performance via over-thinking. The broader claim is that impressive SE-task performance of LLMs coexists with fundamental gaps in consistent terminology use that can affect both LLM consumers and human developers acting on LLM output.

Significance. If the central interpretation holds, the work would be a useful cautionary contribution for the SE and LLM-for-SE communities: it would show that high accuracy on negative (falsified) examples is insufficient evidence of conceptual grasp of standardized vocabulary, and that agentic or reasoning-augmented use does not automatically close the gap. Strengths visible from the design description include grounding in an external, community-standard vocabulary (ISO/IEC/IEEE 24765:2017), a dual semantic/structural falsification scheme, and joint evaluation of accuracy and reasoning-token sensibility. These elements make the study more structured than ad-hoc prompt probes. The practical stakes—miscommunication when LLMs consume or produce SE text—are real. The significance therefore hinges on whether the observed rejection pattern can be cleanly attributed to bias rather than to multi-definitional SE practice, surface cues in the falsifications, or prompt-induced skepticism.

major comments (4)
  1. The headline inference—that high falsified detection plus frequent rejection of correct ISO entries indicates “systematic rejection bias rather than genuine discriminative understanding”—treats each ISO/IEC/IEEE 24765:2017 entry as the unique, unambiguous statement that any model with genuine understanding is obliged to accept. SE terminology is routinely multi-definitional across standards, textbooks, and practice. Without an analysis of (a) how often rejected “correct” items have documented alternative formulations and (b) what the models’ rejection rationales actually cite, the leap from accuracy numbers to “rejection bias” is not secured. This is load-bearing for the central claim and should be addressed by coding rejection reasons and/or testing acceptance of known alternative definitions of the same terms.
  2. The operationalization of “understanding” as classification accuracy on original vs. semantically/structurally falsified definitions, plus sensibility of reasoning tokens, is not shown to rule out shallow surface cues. Key-term substitutions and structural deletions can leave detectable syntactic, distributional, or fluency artifacts that permit high falsified-detection accuracy without conceptual grasp. The manuscript needs either (i) controls that preserve surface statistics while altering meaning, (ii) human baselines on the same items, or (iii) an explicit argument why the chosen falsifications cannot be detected by non-conceptual heuristics. Absent that, the contrast “detects falsified / rejects correct” underdetermines the bias-vs-understanding conclusion.
  3. Scoring whether reasoning tokens “make sense with respect to understanding the definition” introduces a subjective layer that is itself part of the evidence for the central claim. The abstract does not establish inter-annotator agreement, a coding rubric, or blinding. If sensibility judgments are unstable or confounded with fluency, the claim that explicit reasoning “may even hinder performance through over-thinking” rests on a soft measure. A major revision should report agreement statistics, the coding scheme, and sensitivity of the over-thinking conclusion to alternative sensibility criteria.
  4. Prompt framing and decision threshold are not shown to be neutral with respect to acceptance vs. rejection. A prompt that invites verification or skepticism can induce systematic rejection of true statements even when the model’s internal representation of the term is adequate. The manuscript should report the exact prompt templates, any temperature/decoding settings, and an ablation that varies framing (e.g., “is this a valid definition of T?” vs. “does this match the ISO definition of T?”). Without that, rejection rates on correct items cannot be cleanly attributed to model bias rather than task framing.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The four major comments correctly identify load-bearing points for our central claim of systematic rejection bias rather than genuine discriminative understanding of ISO/IEC/IEEE 24765 terminology. We agree that the current manuscript does not yet secure that claim against multi-definitional SE practice, surface-cue alternatives, subjectivity in sensibility scoring, or prompt-framing confounds. We will revise accordingly: code and report rejection rationales; strengthen (and where feasible control) the falsification design; report the sensibility coding scheme with agreement statistics; and publish exact prompts, decoding settings, and a framing ablation. We retain the core experimental design and the cautionary message for SE/LLM-for-SE, but we will qualify the bias interpretation where the new analyses require it. Below we respond point by point.

read point-by-point responses

  1. Referee: The headline inference treats each ISO/IEC/IEEE 24765:2017 entry as the unique, unambiguous statement that any model with genuine understanding is obliged to accept. SE terminology is multi-definitional. Without analysis of (a) alternative formulations of rejected “correct” items and (b) what rejection rationales cite, the leap to “rejection bias” is not secured. Address by coding rejection reasons and/or testing acceptance of known alternative definitions.

    Authors: We agree this is load-bearing and not yet secured. Our operationalization intentionally anchors on ISO/IEC/IEEE 24765:2017 as the community-standard vocabulary for “standardized SE terminology,” which justifies treating its entries as the reference set for this study. That does not, however, establish uniqueness in practice, nor does it show that rejections of correct ISO entries stem from bias rather than legitimate alternative formulations or other rationales. In revision we will (1) systematically code model rejection rationales on correct ISO items (categories such as alternative definition cited, missing nuance, surface wording, over-strictness, hallucination, etc.), with a clear rubric and agreement statistics; (2) for a sample of frequently rejected terms, collect documented alternative definitions from other standards/textbooks and re-test acceptance; and (3) qualify the “rejection bias” claim in light of those results—retaining it only where rejections are not explained by multi-definitional practice or cited alternatives. We will also discuss multi-definitionality as a boundary condition on interpreting ISO-only acceptance as “understanding.” revision: yes

  2. Referee: The operationalization of “understanding” as classification accuracy on original vs. semantically/structurally falsified definitions, plus sensibility of reasoning tokens, is not shown to rule out shallow surface cues. Key-term substitutions and structural deletions can leave syntactic, distributional, or fluency artifacts. Need (i) controls preserving surface statistics while altering meaning, (ii) human baselines, or (iii) an explicit argument why the falsifications cannot be detected by non-conceptual heuristics.

    Authors: This is a fair underdetermination concern. Our dual scheme (key-term substitution; removal of critical information) was intended to force meaning change while preserving definitional form, but we did not demonstrate that high falsified-detection accuracy cannot arise from non-conceptual heuristics. In revision we will: (a) make an explicit, item-level argument for why the chosen falsifications alter definitional content rather than only surface form, including examples and failure modes; (b) add a human baseline on a stratified sample of original and falsified items so that model accuracy and error patterns can be compared to human discriminative performance; and (c) where feasible, introduce a limited control set that better preserves surface statistics (e.g., near-paraphrase or length/fluency-matched distractors) and report whether the detect-falsified / reject-correct pattern persists. We cannot claim a fully cue-free operationalization of “understanding,” and we will state that limitation clearly; the revised evidence will better constrain the bias-vs-understanding interpretation rather than treat accuracy alone as decisive. revision: yes

  3. Referee: Scoring whether reasoning tokens “make sense with respect to understanding the definition” is subjective and part of the evidence for the central claim. The abstract does not establish inter-annotator agreement, a coding rubric, or blinding. If sensibility judgments are unstable or confounded with fluency, the claim that explicit reasoning may hinder performance via over-thinking rests on a soft measure. Report agreement, coding scheme, and sensitivity of the over-thinking conclusion.

    Authors: We agree. Sensibility of reasoning tokens is a soft measure and currently under-specified in the manuscript; the over-thinking claim must not rest on unstable or fluency-confounded judgments. In revision we will: (1) publish the full coding rubric for sensibility (criteria distinguishing conceptual engagement with the definition from fluency, generic skepticism, or irrelevant chain-of-thought); (2) report inter-annotator agreement (e.g., Cohen’s/Fleiss’ κ or percent agreement with adjudication protocol) on a double-coded sample, with blinding to model identity and to gold correctness where possible; (3) separate sensibility from mere fluency in the coding scheme; and (4) re-evaluate the “explicit reasoning may hinder / over-thinking” conclusion under the primary rubric and under at least one alternative sensibility criterion, reporting sensitivity. If agreement is low or the over-thinking effect is not robust, we will soften or retract that claim accordingly. Classification accuracy results will remain primary; sensibility will be clearly labeled as secondary and interpretive. revision: yes

  4. Referee: Prompt framing and decision threshold are not shown to be neutral with respect to acceptance vs. rejection. A verification/skepticism prompt can induce rejection of true statements. Report exact prompt templates, temperature/decoding settings, and an ablation varying framing (e.g., “is this a valid definition of T?” vs. “does this match the ISO definition of T?”). Without that, rejection rates on correct items cannot be cleanly attributed to model bias rather than task framing.

    Authors: We agree that framing is a plausible confound for rejection rates on correct ISO entries and that the manuscript must make prompts and decoding fully transparent. In revision we will: (1) report the exact prompt templates used for all conditions (including any system messages and the decision format); (2) report temperature and other decoding settings, and whether they were held fixed; and (3) run a framing ablation that contrasts at least a verification-style prompt with a more ISO-anchored match prompt (along the lines the referee suggests), and report acceptance/rejection rates on correct vs. falsified items under each framing. We will attribute “systematic rejection bias” to the models only to the extent the pattern survives framing variation; where framing drives rejections, we will restate the finding as framing-sensitive skepticism rather than (or in addition to) intrinsic rejection bias. This directly addresses clean attribution of the central pattern. revision: yes

Circularity Check

0 steps flagged

No significant circularity: empirical classification against external ISO/IEC/IEEE 24765:2017 with constructed negatives; claims do not reduce by construction to fitted inputs or self-defined quantities.

full rationale

This paper is an empirical LLM evaluation study. It takes definitions from the external standard ISO/IEC/IEEE 24765:2017 as ground-truth positives, constructs semantic (key-term substitution) and structural (critical-information removal) falsifications as negatives, prompts models to classify definitions as correct or incorrect, and reports classification accuracy plus a secondary check on whether reasoning tokens make sense. The central observation—high detection of falsified items together with frequent rejection of correct ISO definitions—is a measured outcome on that external benchmark, not a quantity forced by fitting a free parameter to the same data or by defining the target in terms of the result. The interpretive leap to “systematic rejection bias rather than genuine discriminative understanding” is a validity/interpretation judgment about what the accuracy pattern means; it is not a self-definitional reduction (X defined as Y then “derived” as Y), not a fitted-input-called-prediction, and not load-bearing on an unverified uniqueness theorem or ansatz imported from the authors’ own prior work. No equations equate a claimed prediction to its inputs by construction. Self-citation, if any, is not required for the empirical result. Per the analyzer rules, an empirical study self-contained against an external standard with constructed negatives scores 0–2 with empty steps when no specific reduction can be quoted; that is the case here. Concerns that ISO entries may not be unique/unambiguous ground truth, or that falsifications may leave surface cues, are methodological/correctness risks, not circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

Empirical LLM evaluation paper. No fitted physical constants or invented particles. The claim rests on domain assumptions that the ISO vocabulary is the right ground truth, that the two falsification operators produce valid negatives, and that binary classification accuracy plus informal inspection of reasoning tokens measure “understanding.” No free parameters are named in the abstract; any decision thresholds or sampling choices in the full paper are unknown here.

axioms (3)
  • domain assumption ISO/IEC/IEEE 24765:2017 definitions are the authoritative ground truth for correct SE terminology in this evaluation.
    The study labels definitions correct or falsified relative to this standard; if the standard is incomplete, ambiguous, or not what practitioners mean, the rejection-bias interpretation weakens.
  • domain assumption Semantic substitution of key terms and structural removal of critical information yield definitions that a model with genuine understanding should classify as incorrect.
    Core validity of the negative class; poorly chosen substitutions could be still-plausible or trivially broken, changing what accuracy means.
  • ad hoc to paper Classification accuracy on correct vs. falsified definitions, plus whether reasoning tokens “make sense,” is a sufficient operationalization of LLM understanding of SE terminology.
    Understanding is multi-faceted (use in context, generation, multi-turn consistency); the paper’s binary probe may capture rejection bias without capturing full communicative competence.

pith-pipeline@v0.9.1-grok · 6336 in / 2685 out tokens · 46209 ms · 2026-07-08T20:19:41.948878+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) are increasingly used in software engineering (SE), yet there is no systematic study that determines to which degree these LLMs actually understand standardized SE terminology. Lack of such understanding can lead to miscommunication and misunderstanding, both by LLMs consuming text but also by human-developers acting on LLM-generated text. Within this paper, we investigate to which degree state-of-the-art LLMs are able to identify whether definitions from the ISO/IEC/IEEE 24765:2017 Systems and Software Engineering - Vocabulary are correct. We prompt LLMs both with correct definitions, as well as systematically falsified definitions. The falsifications are both semantic (substitution of key terms) and structural (removing critical information). We measure both classification accuracy and whether reasoning tokens generated by the LLMs make sense with respect to understanding the definition. While most LLMs detect falsified definitions with high accuracy, they also reject many correct definitions, indicating a systematic rejection bias rather than genuine discriminative understanding. Explicit reasoning does not consistently improve results and may even hinder performance through over-thinking. Our work demonstrates that while the performance of LLMs (including their agentic use) in many SE tasks is impressive, there are still fundamental issues to understand how this will impact SE, including the consistent use of terminology.

Figures

Figures reproduced from arXiv: 2607.06004 by Fabian C. Pe\~na, Huzaifa Ejaz, Steffen Herbold.

Figure 1
Figure 1. Figure 1: LLM accuracy – True Positive Rate (TPR) on correct definitions [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reasoning quality breakdown for Dfalse. Values are estimated based by extrapolating from the TNR and the CC/CW ratios, and the FPR and the WC/WW ratios reported in Table IV. All LLMs achieved higher detection rates on semantic falsifications than on structural ones (see Table V). Indeed, the semantic falsifications are detected with a TNR of 95.5% or higher, with GPT-5.2-R even reaching 99%. The exception … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 4 internal anchors

  1. [1]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, 2024

    work page 2024

  2. [2]

    Recent advances in natural language processing via large pre-trained language models: A survey,

    B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Comput. Surv., vol. 56, no. 2, Sep. 2023. [Online]. Available: https://doi.org/10.1145/3605943

    work page doi:10.1145/3605943 2023

  3. [3]

    2026), 1–72

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Trans. Softw. Eng. Methodol., vol. 35, no. 2, Jan. 2026. [Online]. Available: https://doi.org/10.1145/3747588

    work page doi:10.1145/3747588 2026

  4. [4]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inThe Twelfth International Conference on Learning Representations, 2024

    work page 2024

  5. [5]

    Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces,

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, J. Shen, G. Ye, H. Lin, J. Poulos, M. Wang, M. Nezhurina, J. Jitsev, D. Lu, O. M. Mastromichalakis, Z. Xu, Z. Chen, Y . Liu, R. Zhang, L. L. Chen, A. Kashyap, J.-L. Uslu, J. Li, J. Wu, M. Yan, S. Bian, V . Sharma, K. Sun, S. Dillmann, A. Ana...

    work page 2026

  6. [6]

    Calero, F

    C. Calero, F. Ruiz, and M. Piattini,Ontologies for software engineering and software technology. Springer Science & Business Media, 2006

    work page 2006

  7. [7]

    Quantifying memorization across neural language models,

    N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang, “Quantifying memorization across neural language models,” inThe Eleventh International Conference on Learning Representations, 2022

    work page 2022

  8. [8]

    Studying memorization of large language models using answers to stack overflow questions,

    L. Caspari, A. Trautsch, M. Granitzer, and S. Herbold, “Studying memorization of large language models using answers to stack overflow questions,”Transactions on Machine Learning Research, 2025

    work page 2025

  9. [9]

    Systems and software engineering — V ocabulary,

    ISO/IEC/IEEE, “Systems and software engineering — V ocabulary,” International Organization for Standardization / International Electrotechnical Commission / Institute of Electrical and Electronics Engineers, Geneva, Switzerland, Standard ISO/IEC/IEEE 24765:2017, Sep. 2017. [Online]. Available: https://www.iso.org/standard/71952.html

    work page 2017

  10. [10]

    Washizaki, Ed.,Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0

    H. Washizaki, Ed.,Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0. IEEE Computer Society,

    work page

  11. [11]

    Available: https://www.swebok.org

    [Online]. Available: https://www.swebok.org

    work page

  12. [12]

    Standard glossary of terms used in software testing,

    ISTQB Board, “Standard glossary of terms used in software testing,”

    work page

  13. [13]

    Available: https://glossary.istqb.org/

    [Online]. Available: https://glossary.istqb.org/

    work page

  14. [14]

    The cpre glossary: An overview of the most important requirements engineering terms v2.2.0,

    M. Glinz, “The cpre glossary: An overview of the most important requirements engineering terms v2.2.0,” 2025. [Online]. Available: https://cpre.ireb.org/en/downloads-and-resources/glossary

    work page 2025

  15. [15]

    Evaluating Large Language Models Trained on Code

    M. Chen, J. Tworek, H. Jun, Q. Yuanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  16. [16]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewskiet al., “Program synthesis with large language models,” 2021

    work page 2021

  17. [17]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,” inThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  18. [18]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yuet al., “Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions,” 2024

    work page 2024

  19. [19]

    Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via LLM,

    C. S. Xia, Y . Deng, and L. Zhang, “Top leaderboard ranking = top coding proficiency, always? evoeval: Evolving coding benchmarks via LLM,” inFirst Conference on Language Modeling, 2024

    work page 2024

  20. [20]

    DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks,

    W. Huang, C. Lee, L. Tng, and S. Ge, “DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks,” 2026. [Online]. Available: https://deepswe.datacurve.ai/

    work page 2026

  21. [21]

    ProgramBench: Can Language Models Rebuild Programs From Scratch?

    J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeveet al., “Programbench: Can language models rebuild programs from scratch?”arXiv preprint arXiv:2605.03546, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  22. [22]

    CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation

    S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tanget al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  23. [23]

    Today was a good day: The daily life of software developers,

    A. N. Meyer, E. T. Barr, C. Bird, and T. Zimmermann, “Today was a good day: The daily life of software developers,”IEEE Transactions on Software Engineering, vol. 47, no. 5, pp. 863–880, 2021

    work page 2021

  24. [24]

    Software developers’ perceptions of productivity,

    A. N. Meyer, T. Fritz, G. C. Murphy, and T. Zimmermann, “Software developers’ perceptions of productivity,” inProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY , USA: Association for Computing Machinery, 2014, p. 19–29

    work page 2014

  25. [25]

    On the feasibility of au- tomated prediction of bug and non-bug issues,

    S. Herbold, A. Trautsch, and F. Trautsch, “On the feasibility of au- tomated prediction of bug and non-bug issues,”Empirical Software Engineering, vol. 25, pp. 5333–5369, 2020

    work page 2020

  26. [26]

    Automated identification and qualitative characterization of safety con- cerns reported in uav software platforms,

    A. Di Sorbo, F. Zampetti, A. Visaggio, M. Di Penta, and S. Panichella, “Automated identification and qualitative characterization of safety con- cerns reported in uav software platforms,”ACM Trans. Softw. Eng. Methodol., vol. 32, no. 3, Apr. 2023

    work page 2023

  27. [27]

    Supporting requirements engineers in recognising security issues,

    E. Knauss, S. Houmb, K. Schneider, S. Islam, and J. J ¨urjens, “Supporting requirements engineers in recognising security issues,” inRequirements Engineering: Foundation for Software Quality, D. Berry and X. Franch, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 4–18

    work page 2011

  28. [28]

    Require- ments classification with interpretable machine learning and dependency parsing,

    F. Dalpiaz, D. Dell’Anna, F. B. Aydemir, and S. C ¸ evikol, “Require- ments classification with interpretable machine learning and dependency parsing,” in2019 IEEE 27th International Requirements Engineering Conference (RE), 2019, pp. 142–152

    work page 2019

  29. [29]

    Code and named entity recognition in StackOverflow,

    J. Tabassum, M. Maddela, W. Xu, and A. Ritter, “Code and named entity recognition in StackOverflow,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 4913–4926

    work page 2020

  30. [30]

    Selu: A software engineering language understanding benchmark,

    F. C. Pe ˜na and S. Herbold, “Selu: A software engineering language understanding benchmark,”arXiv preprint arXiv:2506.10833, 2025

    work page arXiv 2025

  31. [31]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” inProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, T. Linzen, G. Chrupała, and A. Alishahi, Eds. Brussels, Belgium: Association for Computational Lin...

    work page 2018

  32. [32]

    Superglue: A stickier benchmark for general- purpose language understanding systems,

    A. Wang, Y . Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Superglue: A stickier benchmark for general- purpose language understanding systems,” inAdvances in Neural Infor- mation Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019

    work page 2019

  33. [33]

    spacy: Industrial-strength natural language processing in python,

    M. Honnibal, I. Montani, S. V . Landeghem, and A. Boyd, “spacy: Industrial-strength natural language processing in python,” 2020

    work page 2020

  34. [34]

    R. D. Huddleston, G. K. Pullum, and L. Bauer,The Cambridge grammar of the English language. Cambridge University Press Cambridge, 2002

    work page 2002

  35. [35]

    Manning and H

    C. Manning and H. Schutze,Foundations of statistical natural language processing. MIT press, 1999

    work page 1999

  36. [36]

    Glove: Global vectors for word representation,

    J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” inProceedings of the 2014 Conference on Empir- ical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2014, pp. 1532–1543

    work page 2014

  37. [37]

    A Survey on Post-training of Large Language Models

    G. Tie, Z. Zhao, D. Song, F. Wei, R. Zhou, Y . Dai, W. Yin, Z. Yang, J. Yan, Y . Suet al., “A survey on post-training of large language models,” arXiv preprint arXiv:2503.06072, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    Replication kit (anonymized for double- blind review),

    A. Author(s), “Replication kit (anonymized for double- blind review),” https://osf.io/nuath/overview?view only= fb419b4cae064901b852038d862930b4, 2026, URL anonymized for review

    work page 2026