Improving Code Understanding in Large Language Models through Concept-Aware Consistency Learning

ChongYang Liu; Lingxiao Jiang; Qiang Hu; Wei Ma; Xiaoning Ren; Yan Li; Yao Zhang; Yinxing Xue; Yongqiang Lyu

REVIEW 3 major objections 3 minor 43 references

Improving Code Understanding in Large Language Models through Concept-Aware Consistency Learning

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes counterfactual code augmentation plus concept-aware tuning to make LLMs understand data and control flow rather than just mimic code patterns.

desk verdict Plausible idea, zero visible evidence in the abstract—worth a referee's time only if the full paper validates the counterfactuals and benchmarks. read the letter →

arxiv 2508.12620 v2 pith:MXARD2LT submitted 2025-08-18 cs.SE cs.PL

Xiaoning Ren , Qiang Hu , Wei Ma , Chongyang Liu , Yan Li , Yao Zhang , Lingxiao Jiang , Yongqiang Lyu

show 1 more author

Yinxing Xue

This is my paper · ORCID

classification cs.SEcs.PL

keywords largelanguagemodelscodeunderstandingcounterfactualaugmentationconcept-awaretuningconsistencylearningdataflowcontrolprogramcomprehension

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models handle many code tasks well, but they often fail when a task demands real reasoning about how data moves through a program or how control flow branches. This paper tries to fix that by training models on counterfactual code examples: versions of the same program in which a specific concept, such as a data dependency or a loop condition, is deliberately changed. The model is taught to remain consistent with the concept labels during tuning, so it must attend to the underlying structure rather than surface tokens. If the approach works as claimed, code models would become more reliable on reasoning-heavy tasks, which matters for using LLMs in real software engineering. The paper reports that the method improves performance across multiple models and benchmarks.

What carries the argument

The load-bearing mechanism is a training pipeline that pairs each code example with a counterfactual variant and uses concept-aware consistency learning to align the model's behavior with the underlying programming concept. Counterfactual code augmentation supplies the raw material: altered versions of a program in which a selected concept (data flow, control flow, or similar) is modified while unrelated surface details are preserved. Concept-aware consistency learning supplies the training signal: it uses concept labels attached to these augmented examples to push the model toward representations and predictions that are consistent with the intended concept. Together, the augmentation provides examples that differ mainly in one concept, and the consistency objective forces the model to use that concept rather than shallow textual cues.

What would settle it

Train two models identically except for the augmentation: one with the proposed concept-aware counterfactual examples, and one with the same number of counterfactual examples produced by random but syntax-preserving edits such as renaming variables or reordering independent statements. If, on held-out benchmarks that demand data-flow or control-flow reasoning, the concept-aware model does not outperform the random-edit model, then the claim that concept awareness is what drives the gains is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a training framework combining counterfactual code augmentation with concept-aware consistency learning materially improves LLM code understanding. The authors take as their starting point that current models, despite strong benchmark numbers, have only a shallow grasp of fundamental programming concepts such as data flow and control flow. Their proposed remedy is to generate counterfactual variants of code—examples in which one concept is changed while the rest of the program stays as close to the original as possible—and to tune the model so that it learns from these concept-paired examples. The reported result is that this concept-aware tuning changes model behavior enough to produce gains across a range of models and benchmarks, evidence that the models are being guided toward stronger conceptual understanding rather than better pattern memorization.

Load-bearing premise

The paper assumes that the concept labels and counterfactual perturbations it uses actually capture the target programming concepts and that improved benchmark scores reflect genuine conceptual understanding rather than superficial pattern matching.

Editorial extensions

If this is right

The same tuning recipe could be applied on top of existing instruction-tuned code models, improving their performance on tasks that require tracing variables or understanding branches without changing the model architecture.
Benchmarks designed to test deep code reasoning—rather than API recall or style—would be expected to show the largest gains from concept-aware training.
The method could reduce the gap between LLM performance on standard code benchmarks and their performance on edited or adversarial versions of those benchmarks, where shallow models tend to break.
For software engineering practice, a model that genuinely tracks data and control flow would be more trustworthy for code review, bug localization, and automated repair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the central claim holds, the method's success hinges on the quality of concept labels and the validity of the counterfactual perturbations; if those labels are noisy or the perturbations change more than the intended concept, the model could learn spurious correlations. A natural test would be an ablation where counterfactuals are generated by random but syntactically valid edits instead of con
The consistency-learning idea may transfer beyond code: any domain where a few structural concepts control meaning (e.g., logical formulas, configurations, or natural-language entailment) could benefit from the same counterfactual-plus-consistency recipe.
The paper's reported benchmark gains may partly come from the extra training data or regularization that augmentation provides, not from the concept awareness per se; isolating that would require comparing against non-concept augmentation with identical data volume.
If the training works as described, the trained models should show improved performance on code perturbations that preserve syntax but change data or control flow, and should show little or no improvement on perturbations that only change surface naming.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on arXiv:2508.12620. I only have the abstract, so this is a provisional read, but here's where I land.

The problem is real: LLMs handle code fluently yet often fail when data flow and control flow actually matter. That shallow-understanding critique is well supported in the literature. The proposed fix—counterfactual code augmentation plus concept-aware consistency tuning—is a sensible combination of two established ideas, and the twist (making the consistency objective concept-targeted) is plausible enough to be interesting. If it works, it's a useful increment for code LLM reliability. That's the genuine strength: a clear problem, a mechanism with face validity, and a claim that is testable.

Now the soft spots. The abstract gives no numbers, no benchmark names, no model list, no protocol. The central claim rests entirely on "comprehensive evaluation," which is a placeholder, not evidence. More importantly, the whole method depends on two assumptions the abstract doesn't address. First, that the generated counterfactuals preserve program semantics while altering only the targeted concept. If they don't, consistency training could teach models to ignore meaningful structure. Second, that the benchmarks actually require conceptual reasoning rather than surface-pattern matching. Neither assumption is trivial. The stress-test note is on point: without semantic validation of the augmentations and benchmark analysis, the reported gains could just be distributional shortcuts. This isn't a peripheral detail; it's the load-bearing wall of the contribution.

That said, I don't see circularity or invented entities in the abstract. The method description, while thin, is coherent. The absence of prior work citations in the abstract is normal for a cs.SE paper, though it does leave novelty unconfirmed.

Who is this for? Researchers working on code LLM robustness and data augmentation. A reader in that area would want to see the full paper to judge whether the concept definitions are principled or ad hoc. My own verdict: the idea deserves serious evaluation, but the abstract alone is not enough to cite or build on.

Recommendation: send it to peer review. The claims are concrete enough that a competent reviewer can check whether the augmentations are semantically valid and whether the benchmarks measure concept understanding. If the full paper does that, it's a solid contribution; if not, rejection is straightforward. I would not desk-reject, because the core question is worth answering.

One more thing: the abstract overclaims—"demonstrates effectiveness" with no supporting detail. I'd suggest the authors include at least one concrete result and benchmark in the abstract if they want credibility with busy readers.

Best, [Your name]

Referee Report

3 major / 3 minor

Summary. The manuscript as provided consists solely of an abstract. It claims that a counterfactual code augmentation framework combined with concept-aware tuning improves large language models' understanding of fundamental programming concepts such as data flow and control flow, and that a "comprehensive evaluation across multiple models and benchmarks" demonstrates the effectiveness of the approach. No method description, experimental protocol, quantitative results, benchmark names, or model details are included in the text supplied.

Significance. If the claimed results hold, the work could be a valuable contribution to code understanding in LLMs: a training scheme that explicitly targets conceptual structure via counterfactual augmentation, rather than surface form, would address a known weakness of current models. The idea of concept-aware consistency learning is interesting and potentially transferable across models and tasks. However, because the abstract is the only content available, the actual contribution and its validity cannot be assessed. The significance paragraph reflects that the core idea is promising but entirely unverified in the present submission.

major comments (3)

[Abstract / full text (absent)] The submission contains only an abstract; there is no methods section. The proposed counterfactual augmentation framework, the formal definition of concepts such as data flow and control flow, the consistency loss, and the tuning procedure are all unspecified. Without these, the central claim is not evaluable. This is a load-bearing omission: the authors must supply the full technical description before any assessment of soundness is possible.
[Abstract] The statement "comprehensive evaluation across multiple models and benchmarks demonstrates the effectiveness" is unsupported by any numerical results, benchmark names, model identifiers, baselines, or statistical measures. No evidence is presented that the approach outperforms any baseline, nor that any observed gains are consistent or significant. The paper should report concrete metrics with error bars and describe the experimental setup, including datasets, hyperparameters, and comparison methods.
[Abstract] The method's validity rests on two unstated assumptions: (i) the counterfactual perturbations preserve program semantics while altering only the targeted concept, and (ii) the benchmarks used for evaluation actually isolate conceptual reasoning rather than superficial text patterns (e.g., identifier names, formatting, or comment cues). The abstract provides no support for either assumption. The paper should include semantic-equivalence checks for the augmentations (e.g., execution-based equivalence) and an analysis or ablation demonstrating that gains are not attributable to distributional shortcuts.

minor comments (3)

[Abstract] The abstract refers to "studies" that show shallow grasp of programming concepts but gives no citations; please provide references to those studies.
[Abstract] The phrase "comprehensive evaluation" is a strong claim; it would be more informative to name specific benchmarks and report pass@k or accuracy values, and to specify the range of models tested.
[Full text] The full text is missing from the submitted manuscript; the submission appears incomplete and cannot be reviewed in its current form.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity identified in the available text; the abstract's effectiveness claim is an empirical assertion without an accessible derivation chain to inspect.

full rationale

The only substantive text is the abstract, which presents a method (counterfactual code augmentation with concept-aware tuning) and an empirical claim (comprehensive evaluation across multiple models and benchmarks demonstrates effectiveness). No equations, definitions, fitted parameters, or load-bearing self-citations are present, so there is no derivation chain that could reduce to its inputs. The abstract omits details on how concepts are labeled and how counterfactual perturbations are validated, and this is an evidential gap rather than a circular step: a benchmark result could in principle falsify the claim, and the method's parameters are not stated to be fit to the benchmark outcomes. Consistent with the review rules, an incomplete or hard-to-validate evaluation is not treated as circularity without a quotable reduction. No circular step can be exhibited from the provided text.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

None identifiable from the abstract; the method depends on assumed validity of concept definitions and benchmarks, listed as domain assumptions.

assumptions (2)

domain assumption LLMs' grasp of fundamental programming concepts such as data flow and control flow is shallow, motivating the need for concept-aware training.
The abstract asserts this as motivation without citation or empirical support in the abstract.
domain assumption The benchmarks used in the evaluation measure code understanding in a way that reflects the intended concepts.
The abstract says 'multiple models and benchmarks' but does not identify them or justify their validity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Code Understanding in Large Language Models through Concept-Aware Consistency Learning." pith.science (2026). https://pith.science/paper/MXARD2LT

@misc{pith2026250812620,
  author       = {Pith},
  title        = {Pith review of: Improving Code Understanding in Large Language Models through Concept-Aware Consistency Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXARD2LT}},
  note         = {Machine review of arXiv:2508.12620}
}

read the original abstract

Large language models (LLMs) have recently shown impressive results on diverse code-related tasks, benefiting from large-scale training and instruction tuning. However, studies reveal that their grasp of fundamental programming concepts, such as data flow and control flow, remains shallow, leading to fragile performance when code requires deeper reasoning. This limitation restricts the practical adoption of LLMs in real-world software development. To address this issue, this work introduces a counterfactual code augmentation framework combined with concept-aware tuning, designed to guide LLMs toward stronger conceptual understanding. Comprehensive evaluation across multiple models and benchmarks demonstrates the effectiveness of the proposed approach.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 25 canonical work pages

[1]

IEEEtran_HOWTO

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page 2015
[2]

3C6!*S EcN62Z Yyz Zӷ >[:CH., ϳY kfBr6 z=kc §?SJ!̵ mInjlC̻Y[cbz[HkR d

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 2007
[3]

Zhang, B

F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen, ``Repocoder: Repository-level code completion through iterative retrieval and generation,'' arXiv preprint arXiv:2303.12570, 2023

arXiv 2023
[4]

Sch \"a fer, S

M. Sch \"a fer, S. Nadi, A. Eghbali, and F. Tip, ``An empirical evaluation of using large language models for automated unit test generation,'' IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85--105, 2023

work page 2023
[5]

Jiang, K

N. Jiang, K. Liu, T. Lutellier, and L. Tan, ``Impact of code language models on automated program repair,'' in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1em plus 0.5em minus 0.4em IEEE, 2023, pp. 1430--1442

work page 2023
[7]

Orvalho and M

P. Orvalho and M. Kwiatkowska, ``Are large language models robust in understanding code against semantics-preserving mutations?'' arXiv preprint arXiv:2505.10443, 2025

arXiv 2025
[8]

Hooda, M

A. Hooda, M. Christodorescu, M. Allamanis, A. Wilson, K. Fawaz, and S. Jha, ``Do large code models understand programming concepts? a black-box approach,'' arXiv preprint arXiv:2402.05980, 2024

arXiv 2024
[9]

procure, ``procure,'' https://sites.google.com/view/procurecode, 2025

work page 2025

Show all 43 references

[10]

C. A. R. Hoare, ``An axiomatic basis for computer programming,'' Communications of the ACM, vol. 12, no. 10, pp. 576--580, 1969

1969
[11]

F. E. Allen, ``Control flow analysis,'' ACM Sigplan Notices, vol. 5, no. 7, pp. 1--19, 1970

1970
[12]

S. Yang, D. Yan, H. Wu, Y. Wang, and A. Rountev, ``Static control-flow analysis of user-driven callbacks in android applications,'' in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1. 1em plus 0.5em minus 0.4em IEEE, 2015, pp. 89--99

2015
[13]

L. D. Fosdick and L. J. Osterweil, ``Data flow analysis in software reliability,'' ACM Computing Surveys (CSUR), vol. 8, no. 3, pp. 305--330, 1976

1976
[14]

Nilsson-Nyman, G

E. Nilsson-Nyman, G. Hedin, E. Magnusson, and T. Ekman, ``Declarative intraprocedural flow analysis of java source code,'' Electronic Notes in Theoretical Computer Science, vol. 238, no. 5, pp. 155--171, 2009

2009
[15]

Lin and K.-C

J.-C. Lin and K.-C. Wu, ``Evaluation of software understandability based on fuzzy matrix,'' in 2008 IEEE International Conference on Fuzzy Systems (IEEE World Congress on Computational Intelligence). 1em plus 0.5em minus 0.4em IEEE, 2008, pp. 887--892

2008
[16]

P. W. Dart and J. Zobel, ``Efficient run-time type checking of typed logic programs,'' The Journal of Logic Programming, vol. 14, no. 1-2, pp. 31--69, 1992

1992
[17]

Allamanis, E

M. Allamanis, E. T. Barr, S. Ducousso, and Z. Gao, ``Typilus: Neural type hints,'' in Proceedings of the 41st acm sigplan conference on programming language design and implementation, 2020, pp. 91--105

2020
[18]

Jiang, F

J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, ``A survey on large language models for code generation,'' arXiv preprint arXiv:2406.00515, 2024

2024 arXiv
[19]

Verma, V

S. Verma, V. Boonsanong, M. Hoang, K. Hines, J. Dickerson, and C. Shah, ``Counterfactual explanations and algorithmic recourses for machine learning: A review,'' ACM Computing Surveys, vol. 56, no. 12, pp. 1--42, 2024

2024
[20]

W. Ma, S. Liu, W. Wang, Q. Hu, Y. Liu, C. Zhang, L. Nie, and Y. Liu, ``The scope of chatgpt in software engineering: A thorough investigation,'' arXiv preprint arXiv:2305.12138, 2023

2023 arXiv
[21]

Z. Yang, Z. Zhao, C. Wang, J. Shi, D. Kim, D. Han, and D. Lo, ``Unveiling memorization in code models,'' in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1--13

2024
[22]

W. Chen, L. Zhang, L. Zhong, L. Peng, Z. Wang, and J. Shang, ``Memorize or generalize? evaluating llm code generation with evolved questions,'' arXiv preprint arXiv:2503.02296, 2025

2025
[23]

L. Ma, S. Liu, L. Bu, S. Li, Y. Wang, and Y. Liu, ``Speceval: Evaluating code comprehension in large language models via program specifications,'' arXiv preprint arXiv:2409.12866, 2024

2024 arXiv
[24]

Python Software Foundation , ``difflib --- helpers for computing deltas,'' Python Standard Library Documentation , 2023, https://docs.python.org/3/library/difflib.html

2023
[25]

Husain, H.-H

H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, ``Codesearchnet challenge: Evaluating the state of semantic code search,'' arXiv preprint arXiv:1909.09436, 2019

1909 arXiv
[26]

BigCode Project , ``The stack: 3 tb of permissively licensed source code,'' https://huggingface.co/datasets/bigcode/the-stack, 2022, accessed: 2025-05-17

2022
[27]

Hugging Face , ``Codeparrot clean dataset,'' https://huggingface.co/datasets/codeparrot/codeparrot-clean, 2022, accessed: 2025-05-17

2022
[28]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet et al., `` LLaMA 3 : Open foundation and instruction models,'' 2024. [Online]. Available: https://ai.meta.com/llama/

2024
[29]

R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., ``Starcoder: may the source be with you!'' arXiv preprint arXiv:2305.06161, 2023

2023 arXiv
[30]

Roziere, J

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez et al., ``Code llama: Open foundation models for code,'' arXiv preprint arXiv:2308.12950, 2023

2023 arXiv
[31]

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., ``Evaluating large language models trained on code,'' arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[32]

Austin, A

J. Austin, A. Odena, M. Nye, M. Bosma et al., ``Program synthesis with large language models,'' in Advances in Neural Information Processing Systems (NeurIPS), 2021

2021
[33]

Y. Li, D. Choi, J. Chung, S. Narang et al., ``Competition-level code generation with alphacode,'' in International Conference on Learning Representations (ICLR), 2022

2022
[34]

Abdin, J

M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behl et al., ``Phi-3 technical report: A highly capable language model locally on your phone, 2024,'' URL https://arxiv. org/abs/2404.14219, 2024

2024 arXiv
[35]

Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, ``Wizardcoder: Empowering code large language models with evol-instruct,'' arXiv preprint arXiv:2306.08568, 2023

2023 arXiv
[36]

Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang, ``Magicoder: Source code is all you need,'' arXiv preprint arXiv:2312.02120, vol. 10, 2023

2023 arXiv
[37]

Nijkamp, N

E. Nijkamp, N. Kassner, R. Pang, L. Tu, C. Xiong, W. Shi, T. Soru, L. B. Soares, A. Ratner et al., ``A conversational paradigm for program synthesis,'' arXiv preprint arXiv:2203.13474, 2022

2022 arXiv
[38]

T. Y. Zhuo, Z. Yang, Z. Sun, Y. Wang, L. Li, X. Du, Z. Xing, and D. Lo, ``Source code data augmentation for deep learning: A survey,'' arXiv preprint arXiv:2305.19915, 2023

2023 arXiv
[39]

Allamanis, H

M. Allamanis, H. Jackson-Flux, and M. Brockschmidt, ``Self-supervised bug detection and repair,'' in Advances in Neural Information Processing Systems, 2021

2021
[40]

Jain et al., ``Compiler-based data augmentation for code models,'' in Proceedings of the ACM/IEEE International Conference on Software Engineering, 2021

V. Jain et al., ``Compiler-based data augmentation for code models,'' in Proceedings of the ACM/IEEE International Conference on Software Engineering, 2021

2021
[41]

L. Mi, S. Wang, and L. Yang, ``Generating source code variants with auxiliary-classifier gans,'' in ACM Symposium on the Foundations of Software Engineering, 2021

2021
[42]

W. U. Ahmad, S. Chakraborty, and B. Ray, ``Summarise and generate to back-translate: Unsupervised translation of programming languages,'' in EACL, 2023

2023
[43]

X. Dong, J. Li, and Z. Chen, ``Mixcode: Representation-level interpolation for source code models,'' in Proceedings of the 61st Annual Meeting of the ACL, 2023

2023
[44]

W. Ma, S. Liu, M. Zhao, X. Xie, W. Wang, Q. Hu, J. Zhang, and Y. Liu, ``Unveiling code pre-trained models: Investigating syntax and semantics capacities,'' ACM Trans. Softw. Eng. Methodol., vol. 33, no. 7, Aug. 2024. [Online]. Available: https://doi-org.libproxy.smu.edu.sg/10....

2024 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

IEEEtran_HOWTO

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

work page 2015

[2] [2]

3C6!*S EcN62Z Yyz Zӷ >[:CH., ϳY kfBr6 z=kc §?SJ!̵ mInjlC̻Y[cbz[HkR d

11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

arXiv 2007

[3] [3]

Zhang, B

F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen, ``Repocoder: Repository-level code completion through iterative retrieval and generation,'' arXiv preprint arXiv:2303.12570, 2023

arXiv 2023

[4] [4]

Sch \"a fer, S

M. Sch \"a fer, S. Nadi, A. Eghbali, and F. Tip, ``An empirical evaluation of using large language models for automated unit test generation,'' IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85--105, 2023

work page 2023

[5] [5]

Jiang, K

N. Jiang, K. Liu, T. Lutellier, and L. Tan, ``Impact of code language models on automated program repair,'' in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1em plus 0.5em minus 0.4em IEEE, 2023, pp. 1430--1442

work page 2023

[6] [7]

Orvalho and M

P. Orvalho and M. Kwiatkowska, ``Are large language models robust in understanding code against semantics-preserving mutations?'' arXiv preprint arXiv:2505.10443, 2025

arXiv 2025

[7] [8]

Hooda, M

A. Hooda, M. Christodorescu, M. Allamanis, A. Wilson, K. Fawaz, and S. Jha, ``Do large code models understand programming concepts? a black-box approach,'' arXiv preprint arXiv:2402.05980, 2024

arXiv 2024

[8] [9]

procure, ``procure,'' https://sites.google.com/view/procurecode, 2025

work page 2025