Pith. sign in

REVIEW 3 major objections 5 minor 35 references

On the Effect of Token Merging on Pre-trained Models for Code

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding a post-tokenization merge layer cuts code-model FLOPs by up to 19% while largely preserving task performance.

desk verdict A real, measured FLOPs-savings result for post-tokenization merging in code LMs, but the 'semantic unit' grouping is tokenizer-defined and the paper overclaims on retraining and test-set selection. read the letter →

arxiv 2507.14423 v1 pith:JPKZLI77 submitted 2025-07-19 cs.SE

Mootez Saad , Hao Li , Tushar Sharma , Ahmed E. Hassan This is my paper · ORCID
classification cs.SE
keywords tokenmergingsubwordtokenizationbyte-pairencodingpre-trainedcodemodelsFLOPsreductionvulnerabilitydetectiontranslationefficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the token bloat caused by byte-pair encoding for code can be reduced after the fact, inside the model, by merging the hidden representations of subtokens that belong to the same semantic unit. It proposes two merging strategies — a static mean-pooling and a learned attention-weighted combination — both implemented as a drop-in layer that uses the tokenizer's word_ids() groups. Across six pre-trained code models and three software engineering tasks, the merge layer reduces floating-point operations by 1% to 19%, with the largest downstream cost a 1.82-point F1 drop on vulnerability detection and a 2.47-point CodeBLEU gain on code translation. If the result holds, practitioners can save compute on existing models without retraining or changing the tokenizer, and the savings should grow with longer input contexts.

What carries the argument

The load-bearing mechanism is the semantic-unit group computed from the tokenizer's word_ids() mapping, which assigns every subtoken to the whitespace-delimited word it came from. A vectorized algorithm turns these groups into group indices in one pass, and a merging operator — either a simple mean or a softmax attention over a learnable parameter vector — replaces each group's subtoken representations with a single vector. The merged representation is then fed to the remaining transformer layers; the merge layer can be inserted right after the embedding or after chosen intermediate blocks, and the paper picks the best position by Pareto dominance over performance and FLOPs, using the knee point as the final choice.

What would settle it

Take the Big-Vul test set and compare the F1 of the static-mean merge when groups come from word_ids() versus groups aligned to a parser's lexeme boundaries; if whitespace-based merging drops more than the reported 1.82 points whenever parentheses or dots are absorbed into identifier groups, the semantic-unit premise fails.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that subtokens produced by BPE tokenizers can be collapsed back into word-level units — the same grouping that word_ids() exposes — without destroying the representations a pre-trained code model needs. Averaging subtoken vectors before or inside the transformer, or taking a learned attention-weighted sum, shortens the sequence and lowers FLOPs by 1% to 19% across CodeBERT, GraphCodeBERT, UniXCoder, CodeT5, and two CodeT5+ sizes. Downstream performance is largely preserved; classification tasks lose at most about two F1 points, while code translation sometimes improves markedly. The paper further shows that where the merge layer sits matters: late merging (layers 10–12) preserves classification accuracy, whereas early merging works for translation, and a Pareto/knee-point analysis selects the best performance–cost configuration.

Load-bearing premise

The load-bearing assumption is that the tokenizer's word_ids() grouping — which is based on whitespace, not on the actual grammar of the language — corresponds to the 'semantic units' that should be merged; if a merge group cuts across a real syntax boundary, such as joining an opening parenthesis to an identifier, the measured performance preservation may not generalize.

Editorial extensions

If this is right

  • FLOPs drop by 1% to 19% across all six models and three tasks with no architecture or tokenizer change, because sequence length is shortened before or during the forward pass.
  • Classification tasks favor merging late (around layers 10–12), where representations are already contextualized; early merging there discards subtoken detail and hurts F1.
  • Code translation tolerates and even benefits from early merging, with the best CodeBLEU gains reaching 2.47 points over the no-merge baseline.
  • The learned attention-based merge generally beats static averaging, especially in larger encoder-decoder models such as CodeT5+ (770M).
  • For the performance–cost trade-off, the Pareto analysis picks early positions (embedding layer to layer 3) in most cases, except on Big-Vul, where later layers (6–7) are optimal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If merging groups were defined by a parser's lexemes rather than by whitespace-delimited words, the same method could avoid merging across syntax boundaries such as parentheses or dots, likely yielding larger safe savings; this is a natural testable upgrade.
  • Because attention cost grows quadratically, the 1–19% FLOPs range under a 512-token context limit understates what the method would deliver on long-context models; the paper's own bottleneck note points in the same direction.
  • The CodeBLEU gains on translation hint that merging acts as a mild denoiser for generation; an explicit test would compare output diversity or syntactic validity with and without the merge layer.
  • Inspection of the learned attention weights could reveal which subtokens survive merging — for example, whether boundary tokens are preserved — turning the method into an interpretability probe for what code models rely on.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two post-tokenization token-merging strategies for pre-trained code models: a static mean-pooling strategy and a learning-based attention-weighted strategy. Groups are formed from subtokens sharing the same tokenizer word ID, and the merged representations replace the original sequence at a chosen layer position. The authors evaluate the methods on six code models (CodeBERT, GraphCodeBERT, UniXCoder, CodeT5, CodeT5+ 220M, CodeT5+ 770M) across vulnerability detection (Big-Vul), code classification (PoJ-104), and code translation (CodeTrans). They report measured FLOPs reductions of 1–19% with downstream performance generally preserved and, in code translation, improved by up to 2.47 CodeBLEU points. A Pareto analysis is used to recommend optimal layer positions. The main claims are that merging saves computational resources, preserves or enhances performance, and that the best trade-off occurs with early merging and the learning-based strategy.

Significance. If the findings hold, the work offers a simple, drop-in optimization for code transformers that does not require retraining or tokenizer changes, which is practically valuable. The study's strengths include direct FLOPs measurement across six models and three tasks, a systematic sweep over merging positions, and a publicly available replication package. The central FLOPs-saving claim is empirically grounded and internally consistent. However, the significance is tempered by three load-bearing concerns: the operationalization of 'semantic units' via word_ids() rather than code lexemes, the absence of variance or significance reporting for downstream scores, and the selection of Pareto-optimal configurations on the test set. These issues do not invalidate the measured FLOPs reductions, but they affect the generality and the strength of the performance-preservation and optimal-trade-off conclusions.

major comments (3)
  1. [§4.1.3, Algorithm 1] The paper's central premise is that merging targets 'semantic units such as subtokens that form a single identifier' (Section 1, Figure 1). The implementation defines groups from the tokenizer's word_ids() API, which reflects the pre-tokenizer's word segmentation (typically whitespace- and punctuation-based) rather than compiler- or parser-level lexemes. For byte-level BPE tokenizers such as CodeBERT's, snake_case identifiers are split at underscores and punctuation is assigned separate word IDs, so the merged groups are not always single identifiers. The manuscript never verifies how often word_ids groups coincide with identifiers or other lexemes on Big-Vul, PoJ-104, or CodeTrans. Since the measured FLOPs savings and downstream scores depend on the specific grouping, the results should be re-scoped to 'tokenizer-word merging' or supplemented with an alignment analysis (e.g., against tree-sitter identifiers) to support the identifier-based interpretation.
  2. [§5.2, Tables 2–3] All downstream performance numbers are reported as averages over three runs with no variance, confidence intervals, or significance tests. Several differences used to support the 'preserving or even enhancing performance' claim are small relative to typical seed variance (e.g., PoJ-104 learning-based CodeBERT 98.17 vs. baseline 98.32; CodeT5+ 220m learning-based 67.67 vs. 67.41). Without a measure of variability, the reader cannot distinguish genuine preservation from noise. Please report per-seed results or standard deviations, and perform paired tests (or equivalent) for the comparisons that drive the abstract's claims.
  3. [§5.4, Table 3, Figure 7] The Pareto-optimal configuration is selected using test-set metrics: Algorithm 3 takes evaluated configurations (c_i, p_i) where p_i is the F1/CodeBLEU on the test set, and Figure 7 labels the optimal point with its test score. Selecting the best position on the test set and then reporting the same test score as the achieved trade-off is over-optimistic, especially because the differences between candidate positions are often less than one point (Figures 4–6). The selection should be done on the validation split (or with nested cross-validation), and the test set used only once for the finally chosen configuration; at minimum, the bias should be acknowledged and quantified.
minor comments (5)
  1. [Abstract] There is a typo: 'CdoeT5' should be 'CodeT5', and 'Dalhouise' should be 'Dalhousie' in the affiliation.
  2. [§4.1.3] The sentence 'This vectorized implementation reduces the sequence length ... in a parallelized manner' contains a duplicated word: 'leverage leverage GPU parallelism'.
  3. [§5.1] In the RQ1 text, the sentence 'the static merging strategy reduces FLOPs by 15% for CodeBert and CodeBert' should read 'for CodeBert and GraphCodeBert'.
  4. [Figure 1] Figure 1a is rendered in very low resolution and the tokenization example is difficult to read; a higher-resolution version should be provided.
  5. [§5.3] Some figure captions omit the dataset name (e.g., Figure 4 and Figure 5 captions repeat 'Big-Vul' and 'PoJ-104' inside the text but the captions are minimal); adding the dataset and model to each caption would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the empirical claims are benchmarked against external datasets; the only self-citation is non-load-bearing.

full rationale

The paper's central claim is an empirical measurement: merging subtoken representations grouped by HuggingFace word_ids() and evaluating on Big-Vul, PoJ-104, and CodeTrans. The proposed merging operations in Eq. 1-3 are deterministic or trainable aggregation rules, not quantities fitted to reproduce the reported F1/CodeBLEU/FLOPs numbers. Eq. 2's learned vector w is a parameter of the proposed method, trained on the same train splits as the baseline and evaluated on held-out test sets, so it is not a fitted input disguised as a prediction. The FLOPs savings are computed directly from the reduced sequence lengths via fvcore and are a consequence of the number of tokens processed, not a derived result equivalent to the input. The only self-citation is reference [24], used to justify the FLOPs metric and model selection; the FLOPs numbers themselves are measured with fvcore, so this citation is not load-bearing. The word_ids grouping may not coincide with compiler-level lexemes, but that is a construct/external validity concern about whether the measured effect transfers to other grouping definitions, not a circularity: the paper's results are explicitly about its word_ids-based operationalization. No step in the derivation chain reduces to its own inputs.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper makes no first-principles derivation. Its empirical claims rest on the tokenizer's word-ID grouping being semantically meaningful, on fvcore FLOPs being a faithful cost measure, and on hyperparameter transfer across models. The learned attention vector in Eq. 2 is a normal trainable parameter, so no free constants are listed. No new entities are introduced.

assumptions (3)
  • domain assumption The tokenizer's word_ids() output provides valid semantic grouping units for merging.
    Algorithm 1 and Section 4.1.3 group subtokens by word IDs from HuggingFace tokenizers; these IDs are whitespace-based and may merge punctuation into identifiers, so the semantic grouping premise is untested.
  • domain assumption FLOPs counted by fvcore v0.1.5 are an accurate proxy for the computational savings of merging.
    Section 4.2.4 uses fvcore for FLOPs; FLOPs reductions do not necessarily translate into wall-clock or memory savings, especially with variable-length sequences and attention masks.
  • domain assumption Fine-tuning hyperparameters from similar studies [17] transfer to all six models and three tasks.
    Section 7 Internal Validity states the same hyperparameters were used across all experiments; if some models require different learning rates or epochs, the reported merging gains or losses could be confounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Effect of Token Merging on Pre-trained Models for Code." pith.science (2026). https://pith.science/paper/JPKZLI77

@misc{pith2026250714423,
  author       = {Pith},
  title        = {Pith review of: On the Effect of Token Merging on Pre-trained Models for Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPKZLI77}},
  note         = {Machine review of arXiv:2507.14423}
}
abstract

Tokenization is a fundamental component of language models for code. It involves breaking down the input into units that are later passed to the language model stack to learn high-dimensional representations used in various contexts, from classification to generation. However, the output of these tokenizers is often longer than that traditionally used in compilers and interpreters. This could result in undesirable effects, such as increased computational overhead. In this work, we investigate the effect of merging the hidden representations of subtokens that belong to the same semantic unit, such as subtokens that form a single identifier. We propose two strategies: one based on averaging the representations and another that leverages a learning-based approach. Both methods can be seamlessly integrated with existing language models for code. We conduct experiments using six language models for code: CodeBERT, GraphCodeBERT, UniXCoder, CdoeT5, CodeT5+ (220M), and CodeT5+ (770M), across three software engineering tasks: vulnerability detection, code classification, and code translation. Results show that these strategies can reduce the number of floating-point operations by $1\%$ to $19\%$. Regarding downstream performance, the most significant degradation was observed in the vulnerability detection task, where the F1 score decreased by $1.82$ points compared to the baseline. In contrast, for code translation, we observed an improvement of $2.47$ points in CodeBLEU. This work contributes to the broader effort of improving language models for code across multiple dimensions, including both computational efficiency and downstream performance.

Figures

Figures reproduced from arXiv: 2507.14423 by the authors.

Figure 1
Figure 1. Left: A motivating example showing how a Python [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed token merging strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Regression plots of sequence lengths vs. # of subto [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance of CodeBert, GraphCodeBert and UniXCoder on the Big-Vul dataset across all merging strategies and layers. 0 2 4 6 8 10 12 Layer 97.7 97.8 97.9 98.0 98.1 98.2 98.3 98.4 98.5 F1 Score Merging Strategy Mean Learnable No Merge (a) CodeBert 0 2 4 6 8 10 12 Layer…
Figure 5
Figure 5. Figure 5: Performance of CodeBert, GraphCodeBert and UniXCoder on the PoJ-104 dataset across all merging strategies and layers. 0 2 4 6 8 10 12 Layer 0.64 0.66 0.68 0.70 0.72 0.74 CodeBLEU Score Merging Strategy Mean Learnable No Merge (a) CodeT5 (Base) 0 2 4 6 8 10 12 Layer 0.6…
Figure 6
Figure 6. Figure 6: Performance of CodeT5 (Base), CodeT5+ (220m) and CodeT5+ (770m) on the CodeTrans dataset across all merging strategies and layers. On the CodeTrans dataset, merging strategies demonstrate robustness across all layers, frequently outperforming the no￾merge baseline. Acr…
Figure 7
Figure 7. Figure 7: Example of Pareto a front of the CodeBert (mean) configuration on the Big-Vul dataset. Let 𝑠𝑚𝑖𝑛 = (𝑐𝑚𝑖𝑛, 𝑝𝑚𝑖𝑛) be the frontier point with the minimum cost, and let 𝑠𝑚𝑎𝑥 = (𝑐𝑚𝑎𝑥, 𝑝𝑚𝑎𝑥 ) be the point with the maximum performance. The line 𝐿 connecting these two points is…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 18 canonical work pages

  1. [1]

    Anonymous. [n. d.]. Replication package. https://anonymous.4open.science/r/TokenMerger-EFFD

    work page

  2. [2]

    Bonnaerens, Maxim and Dambre, Joni. 2023. Learned thresholds token merging and pruning for vision transformers. Transactions on Machine Learning Research (2023), 16

    work page 2023

  3. [3]

    Costa-jussà, and José A

    Noe Casas, Marta R. Costa-jussà, and José A. R. Fonollosa. 2020. Combining Subword Representations into Word-level Representations in the Transformer Architecture. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , Shruti Rijhwani, Jiang- ming Liu, Yizhong Wang, and Rotem Dror (Eds.). As...

    work page doi:10.18653/v1/2020.acl-srw.10 2020

  4. [4]

    Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. 2024. Knowledge Transfer from High-Resource to Low- Resource Programming Languages for Code LLMs. Proc. ACM Program. Lang. 8, OOPSLA2, Article 295 (Oct. 2024), 32 pages. doi:10.1...

    doi:10.1145/3689735 2024

  5. [5]

    Florian Deissenboeck and Markus Pizka. 2006. Concise and consistent naming. Software Quality Journal 14, 3 (Sept. 2006), 261–282. doi:10.1007/s11219-006- 9219-1

    work page doi:10.1007/s11219-006- 2006

  6. [6]

    Jiahao Fan, Yi Li, Shaohua Wang, and Tien N. Nguyen. 2020. A C/C++ Code Vulnerability Dataset with Code Changes and CVE Summaries. In Proceedings of the 17th International Conference on Mining Software Repositories (MSR’ 20) (Seoul, Republic of Korea). Association for Computing Machinery, New York, NY, USA, 508–512. doi:10.1145/3379597.3387501

    arXiv 2020

  7. [7]

    Siyue Feng, Wenqi Suo, Yueming Wu, Deqing Zou, Yang Liu, and Hai Jin. 2024. Machine Learning is All You Need: A Simple Token-based Approach for Ef- fective Code Clone Detection. In Proceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering (Lisbon, Portugal) (ICSE ’24). Asso- ciation for Computing Machinery, New York, NY, USA, Art...

    arXiv 2024

  8. [8]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 . Association for Computa- tional Linguistics. doi:10.18653/v1/2020.finding...

    doi:10.18653/v1/2020.findings-emnlp.139 2020

Show all 35 references
  1. [9]

    Philip Gage. 1994. A new algorithm for data compression. C Users J. 12, 2 (Feb. 1994), 23–38

    1994

  2. [10]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Smaranda Muresan...

    2022 doi

  3. [11]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Rep...

    2021

  4. [12]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations . https: //openreview.net/forum?id=nZeVKeeFYf9

    2022

  5. [13]

    Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. 2020. Big code != big vocabulary: open-vocabulary models for source code. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (Seoul, South Korea) (ICSE ’2...

    2020

  6. [14]

    Miqing Li, Manuel López-Ibáñez, and Xin Yao. 2023. Multi-objective archiving. IEEE Transactions on Evolutionary Computation (2023)

    2023

  7. [15]

    Siyang Liu, Naihao Deng, Sahand Sabour, Yilin Jia, Minlie Huang, and Rada Mihal- cea. 2023. Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proc...

    2023 doi

  8. [16]

    Xin Liu, Baosong Yang, Dayiheng Liu, Haibo Zhang, Weihua Luo, Min Zhang, Haiying Zhang, and Jinsong Su. 2021. Bridging Subword Gaps in Pretrain- Finetune Paradigm for Natural Language Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Li...

    2021

  9. [17]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shuj...

    2021

  10. [18]

    Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota. 2021. Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. In 2021 IEEE/ACM 43rd International Conference on Soft...

    2021

  11. [19]

    Microsoft Research. 2021. CodeXGLUE: A Benchmark Dataset and Open Chal- lenge for Code Intelligence. https://microsoft.github.io/CodeXGLUE/. Accessed: 2025-07-11

    2021

  12. [20]

    Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016. Convolutional Neu- ral Networks over Tree Structures for Programming Language Processing. Proceedings of the AAAI Conference on Artificial Intelligence 30, 1 (Feb. 2016). doi:10.1609/aaai.v30i1.10139

    2016 doi

  13. [21]

    Vishvak Murahari, Carlos Jimenez, Runzhe Yang, and Karthik Narasimhan. 2022. DataMUX: Data Multiplexing for Neural Networks. In Advances in Neural In- formation Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associ...

    2022

  14. [22]

    Christopher Phelan and Aldo Rustichini. 2015. Pareto Efficiency and Identity . Working Paper. National Bureau of Economic Research. doi:10.3386/w20883

    2015 doi

  15. [23]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE]

    2020 arXiv

  16. [24]

    Mootez Saad, José Antonio Hernández López, Boqi Chen, Dániel Varró, and Tushar Sharma. [n. d.]. An Adaptive Language-Agnostic Pruning Method for Greener Language Models for Code. Proc. ACM Softw. Eng. ([n. d.]). doi:10.1145/ 3715773

  17. [25]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Katrin Erk and Noah A. Smith (Eds.). Associa...

    2016 doi

  18. [26]

    Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, and Dawn Song. 2022. Benchmarking Language Models for Code Syntax Understanding. In Findings of the Association for Computational Linguistics: EMNLP 2022 , Yoav Goldberg, Zor- nitsa Kozareva, and Yue Zhang (Eds.). Association ...

    2022 doi

  19. [27]

    Tree-sitter contributors. 2018. Tree-sitter: An Incremental Parsing System. https: //github.com/tree-sitter/tree-sitter. https://github.com/tree-sitter/tree-sitter Accessed: 2025-07-07

    2018

  20. [28]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. ...

    2017

  21. [29]

    Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi

  22. [30]

    Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Un- derstanding and Generation. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing , Marie-Franc...

    2021 doi

  23. [31]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement De- langue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Can- wen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-A...

    2020

  24. [32]

    Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Ling- ming Zhang. 2022. An extensive study on pre-trained models for program understanding and generation. In Proceedings of the 31st ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis (IS...

    2022

  25. [33]

    Yu Zhao, Lina Gong, Zhiqiu Huang, Yongwei Wang, Mingqiang Wei, and Fei Wu

  26. [2023]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.)

    CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 10...

    2023

  27. [2024]

    In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24)

    Coding-PTMs: How to Find Optimal Code Pre-trained Models for Code Embedding in Vulnerability Detection?. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24) . Association for Computing Machinery, New Yo...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.