Pith. sign in

REVIEW 4 major objections 6 minor 27 references

Modeling Code: Is Text All You Need?

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read IRCoder soft-prompts a frozen code LLM with GNN-encoded LLVM IR graphs, and is claimed to beat both text-only and graph-only baselines on device mapping, algorithm classification, vulnerability detection, and code translation.

desk verdict The new IRGraph format and paired source/IR dataset are real contributions, but the paper never tests whether text alone could deliver the same gains, leaving its title question unanswered. read the letter →

arxiv 2507.11467 v1 pith:RIZKZNWW submitted 2025-07-15 cs.AI cs.SE

classification cs.AIcs.SE
keywords codelanguagemodelsLLVMIRgraphneuralnetworkssoftpromptingcontrolanddataflowtranslationvulnerabilitydetectiondevicemapping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that code LLMs, which reason mainly over surface text, can be given genuine structural awareness of control and data flow without retraining the LLM itself. The proposed approach, IRCoder, compiles each program to LLVM intermediate representation, builds a heterogeneous graph (IRGraph) of values, types, instructions, and attributes, and uses a GNN to project graph and node embeddings into the token-embedding space of a frozen pre-trained LLM as soft prompts. The authors report that the combined model outperforms both the graph-only baseline (ProGraML) and the text-only LLM baseline (Deepseek-Coder-6.7b) on all four benchmarks, with the largest gains on code translation between parallel programming models. If this holds, it offers a parameter-efficient route to structural reasoning in LLMs for tasks such as vulnerability detection and device mapping.

What carries the argument

The machinery has two linked parts. The first is IRGraph, a graph representation of LLVM IR that is finer-grained than prior program graphs: six node types (value, type, size, module, attributes, instruction) and eight edge types (type, dataflow, attribute, CFG, size, symbol, includes, contains), constructed with a Python/llvmlite pipeline and able to represent an entire compilation unit rather than a single function. The second is the GNN-soft-prompt bridge: a two-layer heterogeneous graph convolutional network, pre-trained with masked node-value prediction on roughly two million unlabeled IR files, then fine-tuned so that its mean-pooled graph embedding and per-node embeddings, after a linear projection, are prepended to the source-code token embeddings of a frozen LLM. The mechanism works by making the graph-level and node-level vectors trainable prompt tokens: since the LLM is not updated, the GNN must learn to place structural information where the LLM can attend to it during next-token or classification losses.

What would settle it

Train the same IRCoder pipeline but replace the GNN output with fixed random vectors of the same shape (or with embeddings from a randomly initialized, untrained GNN), keeping everything else identical; if accuracy on DevMap, POJ-104, Juliet, or ParEval does not drop, then the graph's structured content is not what drives the improvement. A complementary check is to randomly permute node orderings while keeping the same multiset of embeddings; a model truly conditioning on structure should be largely insensitive to that permutation.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that graph structure and generative language modeling are not competing representations: a frozen LLM can be made to consume a program's IR graph directly. The paper introduces IRGraph, a heterogeneous graph with six node types and eight edge types that models an entire LLVM compilation unit, and IRCoder, which linearly projects the GNN's graph embedding and per-node embeddings into the LLM's embedding space, forming the input sequence [BOS, G, V1, ..., V|V|, T1, ..., EOS] while the LLM weights stay frozen. Only the GNN is updated during fine-tuning, so the projected vectors must act as interpretable soft prompts for the LLM to use. Reported results show IRCoder matching or beating the graph baseline and consistently beating the text-only LLM baseline across classification and generation benchmarks, and the ablation study attributes most of the gain to value and instruction nodes and to type and dataflow edges.

Load-bearing premise

The load-bearing premise is that a frozen LLM can meaningfully use the prepended GNN output vectors as soft prompts — if the linearly projected graph and node embeddings are not aligned with the LLM's token-embedding space, they would be noise and the reported gains would vanish.

Editorial extensions

If this is right

  • Structural awareness can be added to existing code LLMs without fine-tuning their weights, since only the lightweight GNN is updated during training.
  • On code translation, providing the source function's IR graph raises functional correctness (pass@1), with the largest jump for OpenMP-to-CUDA translation.
  • Because the graph is built from compiler IR and covers whole modules, the approach is applicable to full programs rather than single functions, the usual scope of graph code models.
  • Ablations imply that value and instruction nodes and type and dataflow edges carry most of the benefit, while attribute, size, symbol, includes, contains, and CFG edges add smaller but nonzero contributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication the paper motivates but does not measure is that IRCoder should be more invariant than a text-only LLM to semantics-preserving source rewrites, because the IR graph is identical for equivalent programs.
  • The same frozen-LLM-plus-GNN-soft-prompt recipe could transfer to other structured-input domains, such as compilers, circuit graphs, or scientific data, wherever a GNN can encode structure and the LLM needs to condition on it.
  • A practical limit, which the paper notes but does not test, is context length: prepending one embedding per IR node grows the input with program size, so the method may need graph condensation or sparse node sampling on very large codebases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes IRGraph, a heterogeneous graph representation of LLVM IR with six node types and eight edge types, and IRCoder, a method that prepends GNN-computed graph and node embeddings to the token embeddings of a frozen pretrained LLM as soft prompts. The GNN is pretrained with masked node prediction and then fine-tuned, with the LLM frozen, on a large corpus of source/IR pairs plus synthetic question-answer data. The approach is evaluated on four tasks: DevMap device mapping, POJ-104 algorithm classification, Juliet vulnerability detection, and ParEval parallel code translation, with claims that IRCoder outperforms a text-only DeepSeek-Coder-6.7b baseline and that IRGraph outperforms a ProGraML-style graph baseline. The paper also presents ablation studies removing individual node and edge types from IRGraph.

Significance. If the central claim is substantiated, IRCoder offers a plausible and parameter-efficient way to inject structured program information into large code LLMs without modifying their weights, which would be valuable for compiler-oriented and code-analysis tasks. The strengths of the paper include a thoughtfully designed graph format with finer granularity than prior IR graphs, a large paired source/IR dataset collected via compilation, and evaluation on both classification and generative tasks, with an ablation study of the graph representation. However, the significance currently hinges on an untested attribution: the reported gains of IRCoder over the text-only baseline are claimed to come from the structured graph conditioning, but the experiments do not include a control that provides the same information as plain text. The evaluation also lacks seed variance and a clearly described model-selection protocol, so the quantitative claims are not yet fully supported.

major comments (4)
  1. The central claim that graph structure, rather than additional context or additional trainable parameters, drives IRCoder's improvements is not tested. Every LLM comparison in Section 7 is against DeepSeek-Coder-6.7b prompted with source code only; there is no baseline in which the same LLM receives the same information as ordinary text tokens, for example the serialized LLVM IR text, a textual adjacency list of the IRGraph, or a text description of the graph. Without such a control, the reported gains could be attributed to giving the model more tokens or more parameters, not to the structural conditioning itself. This is load-bearing because the title question, 'Is Text All You Need?', is not answered by the current experiments. Please add at least one text-serialized baseline with comparable information content.
  2. The architecture search is performed per benchmark, but the manuscript does not state whether the selection is made on a validation split or on the test set. If the test set is used to select among GNN architectures, the reported numbers are optimistically biased and the comparison against baselines is not fair. Additionally, all reported accuracies, error rates, and pass@1 scores appear to be single-run numbers with no error bars or seed variance. Please describe the validation protocol and report results over multiple random seeds.
  3. There is an inconsistency in the description of the training procedure. Section 5.2 states that the LLM weights are frozen and only the GNN weights are updated, with the loss being cross entropy for next-token prediction. Section 7.1 states that 'The LLMs are fine-tuned with a classification head instead of a language modeling head.' As written, it is unclear whether the classification head is trained, whether the LLM weights remain frozen for classification tasks, and how the next-token-prediction objective in Section 5.2 is reconciled with a classification head. Please clarify the exact training setup for each of the four benchmarks.
  4. The ablation studies remove node and edge types from IRGraph and train the GNN-only model, but they do not ablate the IRCoder soft-prompt model. Consequently, the results do not establish which graph components are important for the combined model's improvement over the text-only LLM. Since the central claim concerns the value of structured information in the soft-prompt setting, an IRCoder-level ablation (for example, removing the graph embedding G or replacing node embeddings with a pooled vector) would directly test the mechanism.
minor comments (6)
  1. The conclusion states that the proposed models outperform the baseline graph models on all benchmarks, but Section 7.2 reports that IRGraph has the same error rate as ProGraML on POJ-104 (0.03 versus 0.03). Please correct the overstatement.
  2. Using all node embeddings V1,...,V|V| as soft-prompt tokens makes the prompt length proportional to the graph size. The paper does not report the average or maximum number of graph tokens per example, how these fit into the LLM's context window, or whether truncation is used and, if so, which nodes are kept. This information is needed to assess scalability and to rule out information loss due to truncation.
  3. The figures contain the label 'ProgramL' instead of 'ProGraML', and Figure 2 does not clearly indicate which bar corresponds to which model beyond the caption text. Please fix the labels and add a legend.
  4. The Mou et al. 2016 reference is duplicated (2016a and 2016b) with identical titles, and the Juliet test suite citation appears only as a URL without author names. Please normalize the reference list.
  5. There is a typo in the Introduction: 'modern LLMS' should be 'modern LLMs'.
  6. The paper does not state whether the code, the collected paired source/IR dataset, or the trained models will be released. A statement on availability would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is trained and evaluated on independent external benchmarks, and no prediction is fitted to a target derived from the same inputs.

full rationale

The paper's derivation chain is empirical and self-contained. The IRGraph representation (Section 4) is a new graph format with six node types and eight edge types; it is constructed from LLVM IR via llvmlite, not from the evaluation labels. The IRCoder model (Section 5.2) prepends GNN-produced embeddings to frozen DeepSeek-Coder token embeddings; there is no equation in which a reported result is defined by a fitted parameter. Evaluation uses DevMap, POJ-104, and Juliet, which are external datasets, and ParEval (Nichols et al., 2024), which is an externally published benchmark; although two of the current authors are among ParEval's authors, the benchmark is not used to define the method and is falsifiable independently. No uniqueness theorem or ansatz is imported from self-citations; the paper explicitly builds on published GNN-soft-prompting work (Perozzi et al., 2024) and ComPile (Grossman et al., 2024) as data sources. The strongest concern in the skeptical reading is that no baseline gives the same LLM the same IR/graph information as text, so the reported gains are confounded by information quantity and trainable parameters. That is a missing control, not a circular reduction, and it does not meet the evidentiary bar of Eq. X = Eq. Y by construction. Accordingly, there is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about graph sufficiency and soft-prompt alignment, plus hyperparameters selected per benchmark and not fully reported. No new physical or conceptual entities are introduced.

free parameters (3)
  • GNN architecture hyperparameters = Not reported; chosen per benchmark via search
    Section 6.2: 'we conduct an extensive architecture search across various GNN architectures... varying the number of layers and hidden dimensions... done for each benchmark.' Exact values and validation protocol are not given.
  • Fine-tuning learning rate and epochs = 1e-4, 1 epoch
    Section 6.2: fixed for all experiments, not derived from theory; choice may affect results.
  • Masked pretraining node masking ratio = Not reported
    Section 5.1: 'We mask out a random subset of the node values', but the masking fraction is unspecified.
assumptions (4)
  • domain assumption The IRGraph representation faithfully captures the semantics of LLVM IR needed for the evaluated tasks.
    Section 4: six node and eight edge types are assumed sufficient to convey control flow, dataflow, types, attributes, and symbols; if the graph omits important structure, the model cannot recover it.
  • domain assumption A frozen LLM can effectively condition on prepended GNN output vectors that are linearly projected to token embedding space.
    Section 5.2: the input is [BOS, G, V1,...,V|V|, T1,...,EOS] and only GNN weights are updated; this assumes the soft prompt tokens are semantically aligned with the LLM's embedding space.
  • domain assumption Synthetic question-answer pairs generated by GPT-4o are correct and useful for training structural reasoning.
    Section 3.2: 'Questions and answers are generated synthetically using an LLM, namely GPT-4o'; no validation of answer correctness is reported, so training data quality is assumed.
  • standard math Standard GNN message passing and transformer forward computations are correct as described.
    Sections 5.1 and 5.2 rely on standard GCN and attention mechanisms; elementary background is taken for granted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modeling Code: Is Text All You Need?." pith.science (2026). https://pith.science/paper/RIZKZNWW

@misc{pith2026250711467,
  author       = {Pith},
  title        = {Pith review of: Modeling Code: Is Text All You Need?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RIZKZNWW}},
  note         = {Machine review of arXiv:2507.11467}
}
read the original abstract

Code LLMs have become extremely popular recently for modeling source code across a variety of tasks, such as generation, translation, and summarization. However, transformer-based models are limited in their capabilities to reason through structured, analytical properties of code, such as control and data flow. Previous work has explored the modeling of these properties with structured data and graph neural networks. However, these approaches lack the generative capabilities and scale of modern LLMs. In this work, we introduce a novel approach to combine the strengths of modeling both code as text and more structured forms.

Figures

Figures reproduced from arXiv: 2507.11467 by the authors.

Figure 1
Figure 1. Accuracy scores from the DevMap benchmark. Both of the proposed representations outperform the respective baselines. The IRGraph graph representation improves on the ProGraML graph model, while the IRCoder language model builds on the graph to improve the language model. 7.2. Algorithm Classification ProgramL DS-6.7b IRGraph IRCoder 0.0 0.2 0.4 0.6 0.8 1.0 Error Rate 0.03 0.05 0.03 0.02 Algorithm Classification [PI… view at source ↗
Figure 2
Figure 2. Error rate scores from the POJ-104 benchmark. All rep￾resentations are strong at this task. The IRGraph representation scores the same as ProGraML while the IRCoder representation outperforms the Deepseek-Coder baseline. The results on the POJ-104 benchmark are shown in Fig￾ure 2. Here we show the error rate for each representation on the task of classifying the code samples into one of 104 algorithm classes. The pr… view at source ↗
Figure 4
Figure 4. pass@1 scores from the ParEval benchmark comparing Deepseek-Coder and IRCoder. The IRCoder model is better able to translate code when provided with the IR graph during translation. The most pronounced improvement is for the OpenMP to CUDA translation. We observe a substantial increase in correctness of the trans￾lated code between the base LLM and IRCoder. This trend is observed across all translation tasks: sequen… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Ablation study by removing node types from the IR￾Graph representation. We see that value and instruction node types are the most important data points for modeling the IR. The IR attributes are the least important and only reduce the accuracy by less than 1% when remo…
Figure 6
Figure 6. Figure 6: Ablation study by removing edge types from the IRGraph representation. We see that type and dataflow edges are the most important for the model’s performance, while the other edge types have a minimal impact on accuracy. them from the final graph format. 8. Related Wor…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 12 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    URL https://samate.nist.gov/SRD/testsuite.php

    Juliet test suite vl.3, 2017. URL https://samate.nist.gov/SRD/testsuite.php

  3. [3]

    Learning to represent programs with graphs

    Allamanis, M., Brockschmidt, M., and Khademi, M. Learning to represent programs with graphs. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJOFETxR-

  4. [4]

    Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods

    Blau, T., Kimhi, M., Belinkov, Y., Bronstein, A., and Baskin, C. Context-aware prompt tuning: Advancing in-context learning with adversarial methods, 2024. URL https://arxiv.org/abs/2410.17222

  5. [5]

    V., Ben-Nun, T., Hoefler, T., O'Boyle, M

    Cummins, C., Fisches, Z. V., Ben-Nun, T., Hoefler, T., O'Boyle, M. F. P., and Leather, H. Programl: A graph-based program representation for data flow analysis and compiler optimizations. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.\ 224...

  6. [6]

    Talk like a graph: Encoding graphs for large language models, 2023

    Fatemi, B., Halcrow, J., and Perozzi, B. Talk like a graph: Encoding graphs for large language models, 2023. URL https://arxiv.org/abs/2310.04560

  7. [7]

    R., Lee, G

    Gamblin, T., LeGendre, M., Collette, M. R., Lee, G. L., Moody, A., de Supinski, B. R., and Futral, S. The spack package manager: bringing order to hpc software chaos. In SC15: International Conference for High-Performance Computing, Networking, Storage and Analysis, Los Alamitos, CA, USA, nov 2015. IEEE Computer Society. doi:10.1145/2807591.2807623. URL h...

  8. [8]

    S., Diaz, J

    Grossman, A., Paehler, L., Parasyris, K., Ben-Nun, T., Hegna, J., Moses, W. S., Diaz, J. M. M., Trofin, M., and Doerfert, J. Compile: A large IR dataset from production sources. Journal of Data-centric Machine Learning Research, 2024. ISSN XXXX-XXXX. URL https://openreview.net/forum?id=iO9azp1LjQ. Dataset Certification

Show all 27 references
  1. [9]

    K., Clement, C., Drain, D., Sundaresan, N., Yin, J., Jiang, D., and Zhou, M

    Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., Tufano, M., Deng, S. K., Clement, C., Drain, D., Sundaresan, N., Yin, J., Jiang, D., and Zhou, M. Graphcodebert: Pre-training code representations with data flow, 2021. URL htt...

  2. [10]

    Unixcoder: Unified cross-modal pre-training for code representation, 2022

    Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., and Yin, J. Unixcoder: Unified cross-modal pre-training for code representation, 2022. URL https://arxiv.org/abs/2203.03850

  3. [11]

    T., Mahmud, Q

    Jamsaz, A. T., Mahmud, Q. I., Chen, L., Ahmed, N. K., and Jannesari, A. Perfograph: a numerical aware program graph representation for performance optimization and program analysis. In Proceedings of the 37th International Conference on Neural Information Processing Systems, N...

  4. [12]

    Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. CoRR, abs/1609.02907, 2016. URL http://arxiv.org/abs/1609.02907

  5. [13]

    The stack: 3 tb of permissively licensed source code

    Kocetkov, D., Li, R., Ben Allal, L., Li, J., Mou, C., Muñoz Ferrandis, C., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., Bahdanau, D., von Werra, L., and de Vries, H. The stack: 3 tb of permissively licensed source code. Preprint, 2022

  6. [14]

    The power of scale for parameter-efficient prompt tuning, 2021

    Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning, 2021. URL https://arxiv.org/abs/2104.08691

  7. [15]

    Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. CoRR, abs/2101.00190, 2021. URL https://arxiv.org/abs/2101.00190

  8. [16]

    Gpt understands, too, 2023

    Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. Gpt understands, too, 2023. URL https://arxiv.org/abs/2103.10385

  9. [17]

    Liu, Z., He, X., Tian, Y., and Chawla, N. V. Can we soft prompt llms for graph learning tasks? In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, pp.\ 481–484. ACM, May 2024. doi:10.1145/3589335.3651476. URL http://dx.doi.org/10.1145/3589335.3651476

  10. [18]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL http://arxiv.org/abs/1711.05101

  11. [19]

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., Liu, T., Tian, M., Kocetkov, D., Zucker, A., Belkada, Y., Wang, Z., Liu, Q., Abulkhanov, D., Paul, I., Li, Z., Li, W.-D., Risdal, M., Li, J., Zhu, J., Zhuo, T. ...

  12. [20]

    Convolutional neural networks over tree structures for programming language processing

    Mou, L., Li, G., Zhang, L., Wang, T., and Jin, Z. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI'16, pp.\ 1287–1293. AAAI Press, 2016 a

  13. [21]

    Convolutional neural networks over tree structures for programming language processing

    Mou, L., Li, G., Zhang, L., Wang, T., and Jin, Z. Convolutional neural networks over tree structures for programming language processing. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, pp.\ 1287–1293. AAAI Press, 2016 b

  14. [22]

    H., Xie, Z., Rajaram, A., and Bhatele, A

    Nichols, D., Davis, J. H., Xie, Z., Rajaram, A., and Bhatele, A. Can large language models write parallel code? In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing, HPDC '24, New York, NY, USA, 2024. Association for Computi...

  15. [23]

    Gpt-4o system card, 2024

    OpenAI, Hurst, A., and et al. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276

  16. [24]

    Let your graph do the talking: Encoding structured data for llms, 2024

    Perozzi, B., Fatemi, B., Zelle, D., Tsitsulin, A., Kazemi, M., Al-Rfou, R., and Halcrow, J. Let your graph do the talking: Encoding structured data for llms, 2024. URL https://arxiv.org/abs/2402.05862

  17. [25]

    Dataflow analysis-inspired deep learning for efficient vulnerability detection

    Steenhoek, B., Gao, H., and Le, W. Dataflow analysis-inspired deep learning for efficient vulnerability detection. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN...

  18. [26]

    Multitask prompt tuning enables parameter-efficient transfer learning, 2023

    Wang, Z., Panda, R., Karlinsky, L., Feris, R., Sun, H., and Kim, Y. Multitask prompt tuning enables parameter-efficient transfer learning, 2023. URL https://arxiv.org/abs/2303.02861

  19. [27]

    Magicoder: Source code is all you need

    Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.