{"id":"ec44342d-1d34-431a-a49b-6d4f9fc01f20","arxiv_id":"2507.11467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A GNN-encoded LLVM IR graph, prepended as soft prompts to a frozen code LLM, improves accuracy on device mapping, algorithm classification, vulnerability detection, and code translation tasks.","lead":"The authors combine a graph representation of LLVM intermediate representation with a pretrained code language model by feeding the graph through a GNN and inserting its embeddings as soft prompts. On four code benchmarks, the prompted model beats a text-only LLM baseline and a graph-only baseline, suggesting structured code information helps LLMs reason about code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that graph structure, not just extra context, drives IRCoder's gains is untested: no baseline gives the LLM the same IR/graph information as text.","rationale":"I read the paper as attempting to show that injecting an LLVM-IR graph via GNN soft prompts improves code LLMs beyond text-only modeling. The internal mechanism is plausible and the dataset and pretraining effort are real. My main concern is experimental control. The paper's comparisons in Figures 1 through 4 pit IRCoder against a source-text-only LLM and against GNN-only models; they never test an LLM that receives the same graph information as text. This matters because the method adds both new information (the IR graph) and new trainable components (the GNN and projection). Without a text-serialized graph baseline, the reported improvements cannot be attributed to structured fidelity. The reader's weakest_assumption about embedding alignment is a reasonable mechanistic worry, but because the GNN is trained end-to-end with gradients from the frozen LLM, the projection can learn to be useful; the empirical gap is the load-bearing issue. I do not see a fatal flaw; the conclusion should remain conditional pending the control experiment.","tokens_in":11274,"tokens_out":3848,"duration_ms":48852,"concrete_test":"Run the same frozen Deepseek-Coder-6.7b fine-tuning protocol with four prompt conditions on at least DevMap and ParEval: (a) source code only (current baseline), (b) source code plus LLVM IR text, (c) source code plus IRGraph serialized as text (nodes and edges), and (d) IRCoder's soft-prompt graph tokens. If condition (b) or (c) matches or exceeds condition (d), the central claim that structured graph embeddings add value beyond text is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that bridging graph-based analyses into the LLM latent space preserves structured fidelity and outperforms text-only modeling (abstract, Sections 5.2 and 9). But every LLM comparison in Section 7 is against Deepseek-Coder-6.7b prompted with source code only. There is no condition in which the same LLM receives the same information, such as LLVM IR text or a serialized graph adjacency list, as ordinary text tokens. Consequently, the reported IRCoder gains could come entirely from giving the model more tokens or context, or from the trainable GNN and projection adding parameters, rather than from structural conditioning. The title question, 'Is Text All You Need?', is therefore not answered by the experiments. This is a missing control, not an internal contradiction, but it is load-bearing: without it, the improvement attributed to structured modeling is confounded by information quantity and input format. The reader's alignment concern about Section 5.2 is weaker because the GNN and projection are trained end-to-end during soft-prompt fine-tuning, so even initially unaligned embeddings can be driven to carry signal; the absence of a text-serialized baseline cannot be repaired by training.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IRGraph, a heterogeneous graph representation of LLVM IR with six node types and eight edge types, and IRCoder, a method that prepends GNN-computed graph and node embeddings to the token embeddings of a frozen pretrained LLM as soft prompts. The GNN is pretrained with masked node prediction and then fine-tuned, with the LLM frozen, on a large corpus of source/IR pairs plus synthetic question-answer data. The approach is evaluated on four tasks: DevMap device mapping, POJ-104 algorithm classification, Juliet vulnerability detection, and ParEval parallel code translation, with claims that IRCoder outperforms a text-only DeepSeek-Coder-6.7b baseline and that IRGraph outperforms a ProGraML-style graph baseline. The paper also presents ablation studies removing individual node and edge types from IRGraph.","tokens_in":11523,"tokens_out":5360,"duration_ms":69223,"significance":"If the central claim is substantiated, IRCoder offers a plausible and parameter-efficient way to inject structured program information into large code LLMs without modifying their weights, which would be valuable for compiler-oriented and code-analysis tasks. The strengths of the paper include a thoughtfully designed graph format with finer granularity than prior IR graphs, a large paired source/IR dataset collected via compilation, and evaluation on both classification and generative tasks, with an ablation study of the graph representation. However, the significance currently hinges on an untested attribution: the reported gains of IRCoder over the text-only baseline are claimed to come from the structured graph conditioning, but the experiments do not include a control that provides the same information as plain text. The evaluation also lacks seed variance and a clearly described model-selection protocol, so the quantitative claims are not yet fully supported.","major_comments":[{"comment":"The central claim that graph structure, rather than additional context or additional trainable parameters, drives IRCoder's improvements is not tested. Every LLM comparison in Section 7 is against DeepSeek-Coder-6.7b prompted with source code only; there is no baseline in which the same LLM receives the same information as ordinary text tokens, for example the serialized LLVM IR text, a textual adjacency list of the IRGraph, or a text description of the graph. Without such a control, the reported gains could be attributed to giving the model more tokens or more parameters, not to the structural conditioning itself. This is load-bearing because the title question, 'Is Text All You Need?', is not answered by the current experiments. Please add at least one text-serialized baseline with comparable information content.","section":null},{"comment":"The architecture search is performed per benchmark, but the manuscript does not state whether the selection is made on a validation split or on the test set. If the test set is used to select among GNN architectures, the reported numbers are optimistically biased and the comparison against baselines is not fair. Additionally, all reported accuracies, error rates, and pass@1 scores appear to be single-run numbers with no error bars or seed variance. Please describe the validation protocol and report results over multiple random seeds.","section":null},{"comment":"There is an inconsistency in the description of the training procedure. Section 5.2 states that the LLM weights are frozen and only the GNN weights are updated, with the loss being cross entropy for next-token prediction. Section 7.1 states that 'The LLMs are fine-tuned with a classification head instead of a language modeling head.' As written, it is unclear whether the classification head is trained, whether the LLM weights remain frozen for classification tasks, and how the next-token-prediction objective in Section 5.2 is reconciled with a classification head. Please clarify the exact training setup for each of the four benchmarks.","section":null},{"comment":"The ablation studies remove node and edge types from IRGraph and train the GNN-only model, but they do not ablate the IRCoder soft-prompt model. Consequently, the results do not establish which graph components are important for the combined model's improvement over the text-only LLM. Since the central claim concerns the value of structured information in the soft-prompt setting, an IRCoder-level ablation (for example, removing the graph embedding G or replacing node embeddings with a pooled vector) would directly test the mechanism.","section":null}],"minor_comments":[{"comment":"The conclusion states that the proposed models outperform the baseline graph models on all benchmarks, but Section 7.2 reports that IRGraph has the same error rate as ProGraML on POJ-104 (0.03 versus 0.03). Please correct the overstatement.","section":null},{"comment":"Using all node embeddings V1,...,V|V| as soft-prompt tokens makes the prompt length proportional to the graph size. The paper does not report the average or maximum number of graph tokens per example, how these fit into the LLM's context window, or whether truncation is used and, if so, which nodes are kept. This information is needed to assess scalability and to rule out information loss due to truncation.","section":null},{"comment":"The figures contain the label 'ProgramL' instead of 'ProGraML', and Figure 2 does not clearly indicate which bar corresponds to which model beyond the caption text. Please fix the labels and add a legend.","section":null},{"comment":"The Mou et al. 2016 reference is duplicated (2016a and 2016b) with identical titles, and the Juliet test suite citation appears only as a URL without author names. Please normalize the reference list.","section":null},{"comment":"There is a typo in the Introduction: 'modern LLMS' should be 'modern LLMs'.","section":null},{"comment":"The paper does not state whether the code, the collected paired source/IR dataset, or the trained models will be released. A statement on availability would substantially improve reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable candidate for the venue if the missing text-serialized control and the model-selection/variance issues are addressed. Note also that the ParEval benchmark is co-authored by one of the current authors; this is not inappropriate, but the editor may wish to ensure the conflict is disclosed and that the benchmark results are not the sole evidence for the generative claims. The central idea is worth pursuing, but the current evaluation does not yet isolate the effect of structural conditioning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the paper delivers a genuinely new artifact: a fine-grained LLVM IR graph format (six node types, eight edge types), a 2M-file paired source/IR dataset, and the first application of GNN soft prompting to code LLMs. That is real. Second, the central claim—that graph structure, not extra context, explains the gains—is never tested. Every LLM comparison in Section 7 is against DeepSeek-Coder prompted with source code only. There is no condition where the same information (IR text, serialized graph, adjacency list) is fed to the LLM as text tokens. So the gains could come from more tokens, extra parameters, or the trainable GNN/projection, not from structural fidelity.\n\nCredit where it's due: IRGraph is a finer-grained representation than ProGraML or PerfoGraph, and the dataset is a useful resource. The results are consistent across four benchmarks, and the node/edge ablations are informative. The authors are transparent that the soft-prompt mechanism comes from Perozzi et al. and Liu et al.; the novelty is the code-specific graph and scale.\n\nThe missing text-serialized baseline is load-bearing. The paper's title asks 'Is Text All You Need?' and the answer is not addressed because no text-only condition gets the same information. Also, results are single numbers with no error bars; some gains are small, and one benchmark shows IRGraph slightly beating IRCoder. The per-benchmark architecture search is reported without a validation protocol, so there is a risk of test-set overfitting. No code or data released. Minor: ParEval is self-authored by two of the authors, and the synthetic Q&A data is GPT-4o generated; both are disclosed, so acceptable.\n\nBottom line: solid engineering contribution with a fixable experimental gap. It deserves a serious referee, but the right outcome is a major revision that adds a text-serialized IR/graph baseline, error bars, and a concrete architecture-search protocol.\n\nWho is this for? People working on code LLMs and structured code representations. The dataset and graph format are worth knowing about even if the core claim is unproven.\n\nSend it to review. Not a desk reject.","headline":"The new IRGraph format and paired source/IR dataset are real contributions, but the paper never tests whether text alone could deliver the same gains, leaving its title question unanswered.","tokens_in":12073,"tokens_out":3004,"would_cite":true,"duration_ms":31024,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IRCoder soft-prompts a frozen code LLM with GNN-encoded LLVM IR graphs, and is claimed to beat both text-only and graph-only baselines on device mapping, algorithm classification, vulnerability detection, and code translation.","keywords":["code language models","LLVM IR","graph neural networks","soft prompting","control and data flow","code translation","vulnerability detection","device mapping"],"falsifier":"Train the same IRCoder pipeline but replace the GNN output with fixed random vectors of the same shape (or with embeddings from a randomly initialized, untrained GNN), keeping everything else identical; if accuracy on DevMap, POJ-104, Juliet, or ParEval does not drop, then the graph's structured content is not what drives the improvement. A complementary check is to randomly permute node orderings while keeping the same multiset of embeddings; a model truly conditioning on structure should be largely insensitive to that permutation.","tokens_in":11102,"feed_emoji":"⚙️","tokens_out":8305,"duration_ms":76924,"temperature":0.7,"pith_summary":"The paper tries to establish that code LLMs, which reason mainly over surface text, can be given genuine structural awareness of control and data flow without retraining the LLM itself. The proposed approach, IRCoder, compiles each program to LLVM intermediate representation, builds a heterogeneous graph (IRGraph) of values, types, instructions, and attributes, and uses a GNN to project graph and node embeddings into the token-embedding space of a frozen pre-trained LLM as soft prompts. The authors report that the combined model outperforms both the graph-only baseline (ProGraML) and the text-only LLM baseline (Deepseek-Coder-6.7b) on all four benchmarks, with the largest gains on code translation between parallel programming models. If this holds, it offers a parameter-efficient route to structural reasoning in LLMs for tasks such as vulnerability detection and device mapping.","feed_headline":"Graph-soft-prompted code LLM beats text-only and graph-only baselines","feed_subtitle":"IRCoder adds LLVM IR structure as soft prompts to a frozen language model and wins on four code benchmarks.","key_machinery":"The machinery has two linked parts. The first is IRGraph, a graph representation of LLVM IR that is finer-grained than prior program graphs: six node types (value, type, size, module, attributes, instruction) and eight edge types (type, dataflow, attribute, CFG, size, symbol, includes, contains), constructed with a Python/llvmlite pipeline and able to represent an entire compilation unit rather than a single function. The second is the GNN-soft-prompt bridge: a two-layer heterogeneous graph convolutional network, pre-trained with masked node-value prediction on roughly two million unlabeled IR files, then fine-tuned so that its mean-pooled graph embedding and per-node embeddings, after a linear projection, are prepended to the source-code token embeddings of a frozen LLM. The mechanism works by making the graph-level and node-level vectors trainable prompt tokens: since the LLM is not updated, the GNN must learn to place structural information where the LLM can attend to it during next-token or classification losses.","core_discovery":"On its own terms, the paper's central claim is that graph structure and generative language modeling are not competing representations: a frozen LLM can be made to consume a program's IR graph directly. The paper introduces IRGraph, a heterogeneous graph with six node types and eight edge types that models an entire LLVM compilation unit, and IRCoder, which linearly projects the GNN's graph embedding and per-node embeddings into the LLM's embedding space, forming the input sequence [BOS, G, V1, ..., V|V|, T1, ..., EOS] while the LLM weights stay frozen. Only the GNN is updated during fine-tuning, so the projected vectors must act as interpretable soft prompts for the LLM to use. Reported results show IRCoder matching or beating the graph baseline and consistently beating the text-only LLM baseline across classification and generation benchmarks, and the ablation study attributes most of the gain to value and instruction nodes and to type and dataflow edges.","pith_inferences":["A testable implication the paper motivates but does not measure is that IRCoder should be more invariant than a text-only LLM to semantics-preserving source rewrites, because the IR graph is identical for equivalent programs.","The same frozen-LLM-plus-GNN-soft-prompt recipe could transfer to other structured-input domains, such as compilers, circuit graphs, or scientific data, wherever a GNN can encode structure and the LLM needs to condition on it.","A practical limit, which the paper notes but does not test, is context length: prepending one embedding per IR node grows the input with program size, so the method may need graph condensation or sparse node sampling on very large codebases."],"forward_implications":["Structural awareness can be added to existing code LLMs without fine-tuning their weights, since only the lightweight GNN is updated during training.","On code translation, providing the source function's IR graph raises functional correctness (pass@1), with the largest jump for OpenMP-to-CUDA translation.","Because the graph is built from compiler IR and covers whole modules, the approach is applicable to full programs rather than single functions, the usual scope of graph code models.","Ablations imply that value and instruction nodes and type and dataflow edges carry most of the benefit, while attribute, size, symbol, includes, contains, and CFG edges add smaller but nonzero contributions."],"supporting_citations":[{"why":"Supplies the ProGraML graph representation this work extends and the graph baseline and DevMap benchmark it must beat.","marker":"(Cummins et al., 2021)"},{"why":"PerfoGraph's numeric-aware node embeddings are the immediate predecessor the IRGraph design builds on.","marker":"(Jamsaz et al., 2024)"},{"why":"Establishes the GNN-embedding soft-prompt paradigm for encoding graphs into LLMs that IRCoder adapts to compiler IR.","marker":"(Perozzi et al., 2024)"},{"why":"Shows that LLMs can be soft-prompted for graph learning tasks, supporting the feasibility of the frozen-LLM setup.","marker":"(Liu et al., 2024)"},{"why":"Provides the ComPile IR dataset whose files are recompiled and paired with source code to form the ~2M-file training set.","marker":"(Grossman et al., 2024)"},{"why":"The Stack v2 source-code corpus supplies the text snippets that inspire the synthetically generated question-answer data.","marker":"(Lozhkov et al., 2024)"},{"why":"Defines the ParEval benchmark used to measure generative translation correctness, the main test of the combined model.","marker":"(Nichols et al., 2024)"},{"why":"Supplies the POJ-104 algorithm-classification benchmark used to compare graph, LLM, and combined representations.","marker":"(Mou et al., 2016a)"},{"why":"Juliet test suite, the vulnerable/non-vulnerable C++ benchmark on which pair-wise detection accuracy is measured.","marker":"(jul, 2017)"}],"fun_headline_variants":["Soft-prompting LLM with code graphs wins on code tasks","Frozen LLM + IR graph prompts beats text-only baselines","IRCoder: Graph soft prompts for frozen code LLMs","LLM eats LLVM IR: graph prompts beat text alone","Graph-structured soft prompts improve code LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a frozen LLM can meaningfully use the prepended GNN output vectors as soft prompts — if the linearly projected graph and node embeddings are not aligned with the LLM's token-embedding space, they would be noise and the reported gains would vanish.","fun_headline_variants_meta":{"raw":{"variants":["Soft-prompting LLM with code graphs wins on code tasks","Frozen LLM + IR graph prompts beats text-only baselines","IRCoder: Graph soft prompts for frozen code LLMs","LLM eats LLVM IR: graph prompts beat text alone","Graph-structured soft prompts improve code LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1351,"prompt_tokens":825,"completion_tokens":526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":443}},"tokens_in":441,"tokens_out":526,"duration_ms":5470,"temperature":1.0,"reasoning_tokens":443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:07:15.576226+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same IRCoder pipeline but replace the GNN output with fixed random vectors of the same shape (or with embeddings from a randomly initialized, untrained GNN), keeping everything else identical; if accuracy on DevMap, POJ-104, Juliet, or ParEval does not drop, then the graph's structured content is not what drives the improvement. A complementary check is to randomly permute node orderings while keeping the same multiset of embeddings; a model truly conditioning on structure should be largely insensitive to that permutation.","supporting_citations":[{"cited_title":"V., Ben-Nun, T., Hoefler, T., O'Boyle, M","cited_arxiv_id":null,"evidence_quote":"Supplies the ProGraML graph representation this work extends and the graph baseline and DevMap benchmark it must beat."},{"cited_title":"T., Mahmud, Q","cited_arxiv_id":null,"evidence_quote":"PerfoGraph's numeric-aware node embeddings are the immediate predecessor the IRGraph design builds on."},{"cited_title":"S., Diaz, J","cited_arxiv_id":null,"evidence_quote":"Provides the ComPile IR dataset whose files are recompiled and paired with source code to form the ~2M-file training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Stack v2 source-code corpus supplies the text snippets that inspire the synthetically generated question-answer data."},{"cited_title":"H., Xie, Z., Rajaram, A., and Bhatele, A","cited_arxiv_id":null,"evidence_quote":"Defines the ParEval benchmark used to measure generative translation correctness, the main test of the combined model."}],"review_version":1}