Pith. sign in

REVIEW 5 major objections 6 minor 57 references

How Programming Concepts and Neurons Are Shared in Code Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Llama code models translate programming languages by passing through an English-like concept space, with language-exclusive neurons only near the output.

desk verdict First cross-PL map of concept space and neuron sharing; the neuron-layer findings are solid, the English-pivot claim needs tighter controls. read the letter →

arxiv 2506.01074 v1 pith:IBDV3TJ2 submitted 2025-06-01 cs.CL cs.PLcs.SE

Amir Hossein Kargaran , Yihong Liu , François Yvon , Hinrich Schütze This is my paper
classification cs.CLcs.PLcs.SE
keywords codelanguagemodelsprogramminglanguagesmultilingualtranslationlogitlenslanguage-specificneuronscross-lingualalignmentconceptspacemodelinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that programming languages are not stored as isolated systems inside Llama-based code models. During few-shot translation between languages such as Java and Rust, the model's intermediate layers first assign high probability to English tokens and programming-language keywords before committing to the output language, suggesting an English-like conceptual hub. It also claims that neurons in these models follow a layered division of labor: neurons that respond to one language are concentrated in the bottom layers, while neurons exclusive to a single programming language appear in the top layers. For closely aligned languages such as C# and Java, no effective language-specific neurons can be found, because their representations are shared too broadly. If these claims hold, multilingual code generation can be understood as routing through a shared concept space, which has direct consequences for how code models are adapted, compressed, and interpreted.

What carries the argument

The main instrument is the logit lens: at an intermediate layer, the model's unembedding matrix is applied to the hidden state to turn it into token probabilities, revealing which languages' tokens are 'thinkable' before the final prediction. For neuron analysis, the paper uses language activation probability entropy (LAPE), which labels a neuron as language-specific when its activation probability concentrates on one or two languages, and then deactivates those neurons to measure perplexity changes. A third method, MEXA, computes cross-lingual alignment by counting how often parallel code snippets are nearest neighbours in embedding space, identifying which languages sit close to the central concept space.

What would settle it

Translate between two programming languages whose keywords share no English words, or remove all tokens that are simultaneously English words and PL keywords from the keyword sets, and check whether the intermediate-layer English probability peak persists. If it disappears, the English pivot is an artifact of keyword overlap; if it remains for non-overlapping keywords, the concept-space claim is supported.

Watch

Extended reading notes

Core claim

The central claim is that in CodeLlama 7B and Llama 3.1 8B, the internal 'concept space' for programming is English-centric. When the authors decode intermediate-layer embeddings with the logit lens during a few-shot translation task over 21 language pairs, token probabilities and rank scores for English and programming-language keywords rise sharply in the second half of the layers, overtaking the expected output token before the final layer reasserts the correct target token. This indicates that the model passes through an English-like representation even when both the source and target are programming languages. A second claim concerns neuron organization: applying an entropy-based selector to neuron activations over eleven programming languages and English, the authors find that language-specific neurons cluster in the bottom layers, while neurons exclusive to a single programming language concentrate in the top layers near the output. For highly aligned languages such as C# and Java, deactivating their 'specific' neurons also degrades other languages, so their language-specific neurons are not identifiable.

Load-bearing premise

The load-bearing premise is that a token's membership in a keyword list tells you which language the model is 'thinking in,' but many programming-language keywords are ordinary English words, so the English-token signal could be inflated by lexical overlap.

Editorial extensions

If this is right

  • Because English and a few prominent programming languages like C++ and C# sit centrally in the concept space, multilingual code translation could be improved by routing through these hubs, minimizing the distance between source and target languages.
  • The layered neuron pattern—shared and general neurons in bottom layers, language-exclusive neurons in top layers—suggests architectures that share bottom layers across languages and swap only top layers could support new programming languages with minimal added parameters.
  • For closely aligned languages such as Java and C#, shared representations make parameter sharing or adapter-based multilingual support practical, since the model does not need separate neurons for each.
  • The failure to find language-specific neurons for C# and Java means interpretability tools that locate language-specific neurons in natural languages do not transfer cleanly to code languages.
  • The difference between CodeLlama 7B and Llama 3.1 8B implies that a code-heavy training recipe produces more shared, less language-specific neurons, so training data composition directly controls how separable code languages are internally.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the English pivot appears during code translation, then generating code from non-English natural-language comments may also detour through English, meaning code models could inherit the same English-centric behavior documented for natural languages; the paper does not test this directly.
  • Because many programming keywords are ordinary English words, the high English-token probabilities may partly reflect lexical overlap rather than a true conceptual pivot; an experiment with non-overlapping keyword sets would separate these explanations.
  • The layer distribution of language-exclusive neurons suggests a concrete recipe for continual learning: freeze the bottom layers and add only top-layer neurons when teaching a model a new programming language; this is an architectural proposal implied but not stated by the paper.
  • The alignment results predict that the centrality of a programming language in concept space grows with the size of its keyword set and its prevalence in pretraining, so a synthetic language with a deliberately small keyword set should appear peripheral; this is testable with the paper's methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies how two Llama-family models (CodeLlama 7B and Llama 3.1 8B) represent programming languages (PLs) and English in their intermediate layers. Using a super-parallel dataset of code snippets in seven PLs, the authors perform few-shot translation between 21 PL pairs and apply logit lens to measure how often English and PL keyword tokens appear among top decoded tokens at each layer. They report that English tokens, including PL keywords, dominate intermediate representations in the second half of the layers. They also apply MEXA to measure cross-lingual alignment among PLs and LAPE to identify language-specific and exclusive neurons, finding that exclusive neurons tend to appear in top layers and that highly aligned PLs such as C# and Java make language-specific neuron identification difficult. The paper concludes with implications for multilingual code model design and releases code.

Significance. If the findings hold, this is a useful extension of the 'English pivot' hypothesis to the code domain and a first systematic neuron-level analysis of multiple programming languages. The paper leverages established interpretability methods, uses two models, builds a parallel dataset across seven PLs, and makes code available. The observed difference between bottom-layer shared neurons and top-layer exclusive neurons, and the association between cross-lingual alignment and the difficulty of isolating language-specific neurons, are potentially valuable insights. However, the headline claims currently rest on a keyword-based classification prone to lexical confounds and on aggregate curves without statistical support, so the results should be treated with caution until these issues are addressed.

major comments (5)
  1. [§2.1, §2.3, Figure 2] The classification of decoded tokens into 'English' versus 'PL' categories is based on keyword-set membership, and many PL keywords are ordinary English words (e.g., if, for, while, return). The 'English (Excl. PL Keywords)' curve removes only the exact keyword overlap; it does not remove English words that appear in comments or identifiers, which the super-parallel dataset shares across all seven PLs by construction (§2.1). Without a frequency-normalized control against code-corpus token distributions or a control using a non-English natural language, the high English-token probabilities in the second half of layers could reflect lexical overlap and the input surface text rather than an English-centric concept space. The Limitations section explicitly concedes that a precise English-to-PL token dictionary is missing, so this confound is load-bearing for the paper's central claim.
  2. [§3.1–§3.3, Figures 2–7] No error bars, confidence intervals, or significance tests are reported for any empirical quantity in the paper. All conclusions are based on aggregated curves over 581 snippets and 42 translation directions; comparisons such as 'English keywords maintain a high and similar 1/rank value of 0.4' or 'C# achieves the best alignment overall' are presented without variance or significance information. This makes it impossible to assess whether the reported differences between conditions are robust, which is a load-bearing issue for the central empirical claims.
  3. [§3.3, Figure 6] The claim that 'neurons exclusive to each PL tend to appear in the top layers' is not quantitatively established. The text states that 'most language-specific neurons ... are selected from the bottom layers' while 'those that are exclusive to a specific PL are predominantly selected from the top layers,' but Figure 6 is described as showing only exclusive neurons, making the two statements difficult to reconcile. No per-layer counts or statistical comparison of exclusive versus shared neuron distributions are provided, so the abstract's second major finding is not supported by the evidence as presented.
  4. [§3.3] The conclusion that 'identifying language-specific neurons is not feasible' for C# and Java is stronger than the evidence warrants. The results show that with the LAPE method and the chosen hyperparameters (ν=400, τ=0.95), the PPL gap between primary and other languages is smaller for these PLs. This does not demonstrate infeasibility in general; it only indicates that this particular method and setting fail to isolate effective neurons. The claim should be restricted to LAPE under these hyperparameters, or additional identification methods should be tested.
  5. [§2.4, Figure 3] The MEXA alignment scores are computed on the same super-parallel snippets that, as stated in §2.1, share variable names and English comments across all seven PLs. The high alignment of C# and Java with many other languages may therefore be inflated by surface-level lexical overlap rather than representational alignment. This also weakens the subsequent link between alignment and the difficulty of neuron identification. A control that removes comments or uses renamed variables would help ensure the alignment scores reflect semantic alignment rather than shared surface tokens.
minor comments (6)
  1. [§2.3, Figure 2] The definition of the 'probability' and '1/rank' curves is ambiguous: it is not stated whether the values are sums over all tokens in a keyword category or the maximum over those tokens. This should be clarified for reproducibility.
  2. [Table 1] The table uses the symbol '✓?' without explanation, making the row for non-English/PL neuron identification unclear.
  3. [§2.1] The dataset name appears inconsistently as both 'GeeksForGeeks' and 'GeeksForGeeks'; the spelling should be unified.
  4. [§3.1] The text contains a typo: 'rougly' should be 'roughly'.
  5. [Acknowledgements] The heading 'Acknowlegments' is misspelled and should be 'Acknowledgments'.
  6. [§2.5] The FFN equation is missing explicit definitions for the subscript notation in φ(h̃^(ℓ)W1^(ℓ))_r; including these would improve readability.

Circularity Check

3 steps flagged · score 6.0 of 10

The headline 'English closer to concept space' claim is partly forced by classifying overlapping keyword sets and is reinforced by a self-cited MEXA method, though the neuron analysis is largely independent.

  1. self definitional [Section 2.1 and Section 2.3; Figures 2a-2d]
    "We gather, for 22 PLs, programming-specific keywords ... We also extract English keywords from PanLex ... Note that PLs have a limited vocabulary consisting primarily of keywords whereas natural languages have an extensive and continuously evolving lexicon. Additionally, many PLs are influenced by older PLs (Sebesta, 2016), leading to shared structures and common keywords like if, for, while, and return. ..."

    The categories 'English' and 'programming language' are defined by overlapping keyword lists: common PL keywords such as 'if', 'for', 'while', and 'return' count simultaneously as English keywords and as PL keywords. The paper's central observation that 'English and PL keywords overtake the expected tokens at first' and that 'the concept space is closer to English (including PL keywords)' is therefore guaranteed in part by the labeling scheme. The comparison between 'English Keywords' and 'English (Excl. PL Keywords)' removes exact keyword overlap but does not remove the shared surface tokens in the super-parallel snippets (consistent variable names and comments in English) or the unembedding prior over frequent code tokens.

  2. self citation load bearing [Section 2.4, Method 2; Section 3.2; Table 1]
    "We employ MEXA (Kargaran et al., 2024), a measure of cross-lingual alignment, to determine which PL aligns most closely with the majority of the selected PLs in the model's intermediate layers. ... The high alignment of C# and C++ further supports the influence of popular PLs, as discussed in Section 3.1."

    MEXA is cited to prior work by the same first-author group (Kargaran et al., 2024) and is then used to interpret the same hidden states that the paper's keyword analysis already claims are 'closer to English (including PL keywords)'. The paper does not derive the MEXA alignment score from the logit-lens keyword probabilities, but it uses the two methods to mutually reinforce the same story: C# is both 'one of the largest keyword sets appearing frequently in intermediate layers' and 'the best-aligned PL overall'. This is not a uniqueness theorem, but it makes the cross-lingual alignment section a self-citation that amplifies the first claim rather than an independent test.

1 more flagged steps
  1. other [Limitations, Section 6]
    "Second, while we use keywords to interpret latent embeddings, a more precise approach would involve constructing a dictionary mapping PL keywords to each other and their English equivalents. However, this is not always feasible, as some PL keywords lack direct English meanings or map to multiple tokens."

    The limitation explicitly concedes that the keyword classification is a proxy without a precise mapping to English equivalents. Since the main evidence for 'concept space is closer to English' depends on this proxy, the paper's own limitation statement shows that the central claim does not yet have an independent test. The absence of a dictionary means the paper cannot distinguish between the model genuinely detouring through English representations and the model merely preferring surface tokens that are English words because they are also code vocabulary or appear in shared comments. This concession does not make the neuron analysis circular, but it does weaken the load-bearing first claim.

full rationale

The paper's core neuron-level findings (LAPE-based language-specific neuron localization, bottom-layer concentration, top-layer exclusive neurons, difficulty for aligned PLs like C#/Java) are genuinely empirical, threshold-dependent, externally evaluated via PPL changes, and not fitted to the conclusion. The cross-lingual alignment measure MEXA is a well-defined, parameter-free score and its use is not circular in itself; the self-citation only becomes mildly load-bearing because the same group's prior method is used to reinforce the same English-centric story. The main circularity is in Method 1: the classification of decoded tokens into 'English' versus 'PL' categories uses keyword sets that overlap by construction, and the super-parallel translation input itself contains English comments and identifiers. The paper even concedes in the Limitations that a precise English-to-PL dictionary is missing. Therefore the headline 'concept space is closer to English' is not fully forced by definition, but its measurement is inflated by lexical overlap and shared surface English, so a score of 6 is appropriate: one or more 'predictions' reduce partially by construction. The neuron analysis and MEXA findings supply independent content, preventing a score of 8 or higher.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper contributes an empirical analysis built on established interpretability tools. Its central claims rest on several interpretive assumptions (logit lens as a window into concept space, keyword membership as language identity, GeeksforGeeks snippets as semantically parallel, MEXA and LAPE as valid measures). These are plausible domain assumptions from prior literature, not free mathematical constants, but they are load-bearing: if any fails, the corresponding conclusions weaken. The only tuned quantities are LAPE selection thresholds and the logit-lens token budget, which are disclosed.

free parameters (3)
  • tau (activation quantile for LAPE) = 0.95
    After default from Tang et al. (2024); determines activation probability thresholds 0.531 (CodeLlama 7B) and 0.554 (Llama 3.1 8B) in Section 2.5. Different tau changes which neurons are selected.
  • nu (number of language-specific neurons selected per language) = 400, varied up to 15,000
    Section 2.5 and Figure 5; choosing the same nu per language makes comparisons easier but is an arbitrary equalization that affects PPL-change curves.
  • alpha (top decoded tokens tracked by logit lens) = 10
    Section 2.3; only the top 10 tokens are classified as English or PL, so rarer language tokens are never counted.
assumptions (5)
  • domain assumption Logit lens decoding of intermediate hidden states with the unembedding matrix approximates the model's internal concept space.
    Method 1 (Section 2.3) interprets all probability curves as evidence about the concept space; this is an established but non-proven interpretive assumption.
  • domain assumption A token's membership in a PL keyword list or an English dictionary is a reliable indicator of the language identity the model is 'thinking in'.
    Used throughout Sections 2.1 and 3.1; many PL keywords are English words, so the assumption can overstate English centrality.
  • domain assumption The 581 GeeksforGeeks snippets are semantically parallel across all seven programming languages, including identical variable names and comments.
    Section 2.1; alignment and few-shot translation probes depend on this equivalence.
  • domain assumption MEXA alignment based on cosine similarity of position-weighted averaged embeddings measures cross-lingual alignment.
    Section 2.4; borrowed from Kargaran et al. (2024), but the link from embedding similarity to 'alignment' is interpretive.
  • domain assumption LAPE entropy over neuron activation probabilities identifies functionally language-specific neurons.
    Section 2.5; the neuron selection depends on thresholds and an entropy criterion, and the causal PPL test for CodeLlama shows weak effects for most PLs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Programming Concepts and Neurons Are Shared in Code Language Models." pith.science (2026). https://pith.science/paper/IBDV3TJ2

@misc{pith2026250601074,
  author       = {Pith},
  title        = {Pith review of: How Programming Concepts and Neurons Are Shared in Code Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBDV3TJ2}},
  note         = {Machine review of arXiv:2506.01074}
}
read the original abstract

Several studies have explored the mechanisms of large language models (LLMs) in coding tasks, but most have focused on programming languages (PLs) in a monolingual setting. In this paper, we investigate the relationship between multiple PLs and English in the concept space of LLMs. We perform a few-shot translation task on 21 PL pairs using two Llama-based models. By decoding the embeddings of intermediate layers during this task, we observe that the concept space is closer to English (including PL keywords) and assigns high probabilities to English tokens in the second half of the intermediate layers. We analyze neuron activations for 11 PLs and English, finding that while language-specific neurons are primarily concentrated in the bottom layers, those exclusive to each PL tend to appear in the top layers. For PLs that are highly aligned with multiple other PLs, identifying language-specific neurons is not feasible. These PLs also tend to have a larger keyword set than other PLs and are closer to the model's concept space regardless of the input/output PL in the translation task. Our findings provide insights into how LLMs internally represent PLs, revealing structural patterns in the model's concept space. Code is available at https://github.com/cisnlp/code-specific-neurons.

Figures

Figures reproduced from arXiv: 2506.01074 by the authors.

Figure 1
Figure 1. Illustration of logit lens (Nostalgebraist, 2020) applied to CodeLlama 7B for the task of translating a for loop from Java to Rust (showing only Rust loop here). The y-axis shows layers, the x-axis input to￾kens, and color next-token probabilities (red: low, blue: high). Terms decoded in intermediate layers, such as interval, range, until, and ten, are not keywords in Java or Rust but belong to other PLs (Python, Go… view at source ↗
Figure 2
Figure 2. Language keyword probability or 1 rank value (best keyword rank) during translation task. The PLs contributing the most to each score, selected from the 22 PL keywords, are C++ and C#. 3 Results 3.1 Method 1: Interpreting latent embeddings We present the results of interpreting latent embed￾dings for the translation task in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. MEXA alignment score. The minimum value of the MEXA alignment score is 0. The figures are limited to scores above 0.4 for better visualization. overall across all layers in both models, though the difference between C-family PLs and Java is minimal. Both models show fewer alignments for Python. JavaScript is the best-aligned PL for both PHP and Python. The high alignment of C# and C++ further supports the influence … view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Impact of LAPE neuron identification. X-axis: Number of shared neurons for each language. Y-axis: [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Number of “language-specific” neurons, i.e., [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Impact of LAPE neuron identification. X-axis: Number of shared neurons for each language. Y-axis: [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 22 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2408.10914 To code, or not to code? exploring impact of code in pre-training . Preprint, arXiv:2408.10914

    arXiv 2024

  4. [4]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. https://arxiv.org/abs/2303.08112 Eliciting latent predictions from transformers with the tuned lens . Preprint, arXiv:2303.08112

    arXiv 2023

  5. [5]

    Sunit Bhattacharya and Ond r ej Bojar. 2023. https://doi.org/10.18653/v1/2023.blackboxnlp-1.9 Unveiling multilinguality in transformer models: Exploring language specificity in feed-forward networks . In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 120--126, Singapore. Association for Computational...

    doi:10.18653/v1/2023.blackboxnlp-1.9 2023

  6. [6]

    Sunit Bhattacharya and Ondřej Bojar. 2024. https://arxiv.org/abs/2404.13855 Understanding the role of ffns in driving multilingual behaviour in llms . Preprint, arXiv:2404.13855

    work page Pith review arXiv 2024

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  8. [8]

    CodeParrot. 2022. https://hf.co/datasets/codeparrot/github-code Git H ub code dataset

    work page 2022

Show all 57 references
  1. [9]

    Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024. https://arxiv.org/abs/2406.11931 Deep S eek- C oder-v2: Breaking the barrier of closed-source models in code intelligence . Preprint...

    2024 arXiv

  2. [10]

    Khapra, Anoop Kunchukuttan, and Pratyush Kumar

    Sumanth Doddapaneni, Gowtham Ramesh, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2021. https://arxiv.org/abs/2107.00676 A primer on pretrained multilingual language models . Preprint, arXiv:2107.00676

    2021 arXiv

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  4. [12]

    Cl \'e ment Dumas, Veniamin Veselovsky, Giovanni Monea, Robert West, and Chris Wendler. 2024. https://openreview.net/forum?id=0ku2hIm4BS How do llamas process multilingual text? a latent exploration through activation patching . In ICML 2024 Workshop on Mechanistic Interpretability

    2024

  5. [13]

    G \'a llego, Ioannis Tsiamas, and Marta R

    Javier Ferrando, Gerard I. G \'a llego, Ioannis Tsiamas, and Marta R. Costa-juss \`a . 2023. https://doi.org/10.18653/v1/2023.acl-long.301 Explaining how transformers use context to build predictions . In Proceedings of the 61st Annual Meeting of the Association for Computatio...

    2023 doi

  6. [14]

    GeeksforGeeks . 2008. https://www.geeksforgeeks.org/ Geeksforgeeks: A computer science portal for geeks

    2008

  7. [15]

    Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.3 Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space . In Proceedings of the 2022 Conference on Empirical Methods in Natural L...

    2022 doi

  8. [16]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.446 Transformer feed-forward layers are key-value memories . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484--5495, ...

    2021 doi

  9. [17]

    Git H ut. 2024. https://madnight.github.io/githut Git H ut 2.0: Language popularity in G it H ub repositories

    2024

  10. [18]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. https://arxiv.org/abs/2401.14196 Deep S eek- C oder: When the large language model meets programming -- the rise of ...

    2024 arXiv

  11. [19]

    Muhammad Umair Haider, Umar Farooq, A. B. Siddique, and Mark Marron. 2024. https://arxiv.org/abs/2407.04868 Looking into black box code language models . Preprint, arXiv:2407.04868

    2024 arXiv

  12. [20]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1--79

    2024

  13. [21]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. https://arxiv.org/abs/2403.07974 Live C ode B ench: Holistic and contamination free evaluation of large language models for code . Preprin...

    2024 arXiv

  14. [22]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. https://arxiv.org/abs/2406.00515 A survey on large language models for code generation . Preprint, arXiv:2406.00515

    2024 arXiv

  15. [23]

    David Kamholz, Jonathan Pool, and Susan Colowick. 2014. https://aclanthology.org/L14-1023/ P an L ex: Building a resource for panlingual lexical translation . In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC `14) , pages 3145--31...

    2014

  16. [24]

    Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, and Hinrich Schütze. 2024. https://arxiv.org/abs/2410.05873 MEXA : Multilingual evaluation of english-centric LLM s via cross-lingual alignment . Preprint, arXiv:2410.05873

    2024 arXiv

  17. [25]

    Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. 2024. https://doi.org/10.18653/v1/2024.naacl-long.384 On the multilingual ability of decoder-based pre-trained language models: Finding and controlling language-specific neurons . In Proceedings ...

    2024 doi

  18. [26]

    Marie-Anne Lachaux, Baptiste Rozière, Lowik Chanussot, and Guillaume Lample. 2020. https://arxiv.org/abs/2006.03511 Unsupervised translation of programming languages . Preprint, arXiv:2006.03511

    2020 arXiv

  19. [27]

    Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, and 1 others. 2023. Starcoder: may the source be with you! Transactions on Machine Learning Research

    2023

  20. [28]

    Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. 2021. https://doi.org/10.18653/v1/2021.acl-long.25 Learning language specific sub-network for multilingual machine translation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1...

    2021 doi

  21. [29]

    Weize Liu, Yinlong Xu, Hongxia Xu, Jintai Chen, Xuming Hu, and Jian Wu. 2024 a . https://doi.org/10.18653/v1/2024.emnlp-main.662 Unraveling babel: Exploring multilingual activation patterns of LLM s and their applications . In Proceedings of the 2024 Conference on Empirical Me...

    2024 doi

  22. [30]

    Yihong Liu, Runsheng Chen, Lea Hirlimann, Ahmad Dawar Hakimi, Mingyang Wang, Amir Hossein Kargaran, Sascha Rothe, François Yvon, and Hinrich Schütze. 2025. https://arxiv.org/abs/2502.17355 On relation-specific neurons in large language models . Preprint, arXiv:2502.17355

    2025

  23. [31]

    Yue Liu, Chakkrit Tantithamthavorn, Yonghui Liu, and Li Li. 2024 b . On the reliability and explainability of language models for program generation. ACM Transactions on Software Engineering and Methodology, 33(5):1--26

    2024

  24. [32]

    Michael R Lyu, Baishakhi Ray, Abhik Roychoudhury, Shin Hwei Tan, and Patanamon Thongtanunam. 2024. Automatic programming: Large language models and beyond. ACM Transactions on Software Engineering and Methodology

    2024

  25. [33]

    Nick Meyer and Leigh McCulloch. 2022. Keywords: A list and count of keywords in programming languages. https://github.com/e3b0c442/keywords

    2022

  26. [34]

    Ahmad Haji Mohammadkhani, Chakkrit Tantithamthavorn, and Hadi Hemmatif. 2023. Explaining transformer-based code models: What do they learn? when they do not work? In 2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 96--106. IEEE

    2023

  27. [35]

    Aaron Mueller, Yu Xia, and Tal Linzen. 2022. https://doi.org/10.18653/v1/2022.conll-1.8 Causal analysis of syntactic agreement neurons in multilingual language models . In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 95--109, Abu...

    2022 doi

  28. [36]

    Niklas Muennighoff. 2022. https://arxiv.org/abs/2202.08904 Sgpt: Gpt sentence embeddings for semantic search . Preprint, arXiv:2202.08904

    2022 arXiv

  29. [37]

    Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Code G en: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474

    2022 arXiv

  30. [38]

    Nostalgebraist. 2020. Interpreting GPT : The logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  31. [39]

    Matteo Paltenghi and Michael Pradel. 2021. Thinking like a developer? comparing the attention of humans with neural models of code. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 867--879. IEEE

    2021

  32. [40]

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. 2025. https://arxiv.org/abs/2501.01257 Code E lo: Benchmarking co...

    2025 arXiv

  33. [41]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code L lama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  34. [42]

    Robert W. Sebesta. 2016. Concepts of Programming Languages, 11 edition. Pearson Education Limited, Harlow, England

    2016

  35. [43]

    Noam Shazeer. 2020. https://arxiv.org/abs/2002.05202 Glu variants improve transformer . Preprint, arXiv:2002.05202

    2020 arXiv

  36. [44]

    Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. https://doi.org/10.18653/v1/2024.acl-long.309 Language-specific neurons: The key to multilingual capabilities in large language models . In Proceedings of the 62nd...

    2024 doi

  37. [45]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . https://arxiv.org/abs/2302.13971 Lla...

    2023 arXiv

  38. [46]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023 b . https://arxiv.org/abs/2307.09288 Llama 2: Open foundation and fine-tuned chat models . Preprint, a...

    2023 arXiv

  39. [47]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  40. [48]

    Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2022. What do they capture? a structural analysis of pre-trained language models for source code. In Proceedings of the 44th International Conference on Software Engineering, pages 2377--2388

    2022

  41. [49]

    Weixuan Wang, Barry Haddow, Minghao Wu, Wei Peng, and Alexandra Birch. 2024. https://arxiv.org/abs/2406.09265 Sharing matters: Analysing neurons across languages and tasks in llms . Preprint, arXiv:2406.09265

    2024

  42. [50]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. https://arxiv.org/abs/2402.10588 Do llamas work in english? on the latent language of multilingual transformers . Preprint, arXiv:2402.10588

    2024 arXiv

  43. [51]

    Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. 2025. https://openreview.net/forum?id=FrFQpAgnGE The semantic hub hypothesis: Language models share semantic representations across languages and modalities . In International Conference on Learning Repre...

    2025

  44. [52]

    Wanying Xie, Yang Feng, Shuhao Gu, and Dong Yu. 2021. https://doi.org/10.18653/v1/2021.acl-long.445 Importance-based neuron allocation for multilingual neural machine translation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and t...

    2021 doi

  45. [53]

    Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1--10

    2022

  46. [54]

    Biao Zhang, Ankur Bapna, Rico Sennrich, and Orhan Firat. 2021. Share or not? learning to schedule language-specific capacity for multilingual translation. In Ninth International Conference on Learning Representations 2021

    2021

  47. [55]

    Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. https://openreview.net/forum?id=ctXYOoAgRy How do large language models handle multilingualism? In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  48. [56]

    Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022 a . https://arxiv.org/abs/2206.08474 Xlcost: A benchmark dataset for cross-lingual code intelligence . Preprint, arXiv:2206.08474

    2022 arXiv

  49. [57]

    Ming Zhu, Karthik Suresh, and Chandan K Reddy. 2022 b . Multilingual code snippets training for program translation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11783--11790

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.