Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Finetuning Lightweight LLMs for Control Flow Graph Generation

T0 review · 4 major / 5 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned lightweight LLMs generate control-flow graphs even from incomplete or erroneous code, and transfer across languages.

desk verdict Solid lightweight multi-model fine-tuning study for CFG digraphs on LeetCode, with a real but unvalidated shared-label assumption under the robustness claim. read the letter →

arxiv 2607.04582 v1 pith:UU6WFNTC submitted 2026-07-06 cs.SE

Hanyu Zhang , Tomoji Kishi This is my paper
classification cs.SE
keywords controlflowgraphlargelanguagemodelfinetuningcodegenerationlightweightLLMserror-augmentedcross-languagegeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Traditional tools for building control flow graphs need complete, compilable, syntax-clean code and usually depend on language-specific parsers. This paper shows that small language models, after supervised fine-tuning on a unified graph format, can produce usable CFGs from ordinary source snippets, including ones that contain typos, missing symbols, or unfinished statements. The authors define a compact digraph-style output, build training data from LeetCode solutions in Java and Python with automatic AST-to-CFG labels plus deliberate error variants that keep the same intended control flow, and fine-tune six models around 3–7B parameters. The best models reach high node-level F1 and still work when code is imperfect; the same models retain partial ability on JavaScript, a language never seen in training. A sympathetic reader cares because many real development and analysis settings involve partial or broken code for which classical CFG tools simply fail.

What carries the argument

A compact, Graphviz-inspired digraph schema (method name, typed nodes with statement labels, directed edges) plus a task-specific instruction prompt that forces models to emit only that schema; training uses Tree-sitter AST-to-CFG labels and error-augmented copies that share the same target CFG.

What would settle it

On a held-out set of real incomplete or buggy snippets whose ground-truth CFGs are independently verified by human inspection or a different tool, measure whether node and edge F1 stay within a few points of clean-code performance; a large drop falsifies the robustness claim.

Watch

Extended reading notes

Core claim

After fine-tuning on a unified digraph CFG representation and a fixed prompt, lightweight LLMs (both code-specific and general-purpose) generate structurally useful control-flow graphs from source code, remain nearly as accurate on deliberately erroneous variants as on clean code, and exhibit measurable cross-language transfer to an unseen language (JavaScript).

Load-bearing premise

The injected syntax and incompleteness errors are assumed never to change the intended control-flow structure, so each broken snippet can safely reuse the clean code’s CFG label.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes fine-tuning lightweight LLMs (CodeLlama-7B, Qwen2.5-Coder-7B, DeepSeek-Coder-6.7B, Llama-3.2-3B, Qwen3-4B, Phi-4-mini) for Control Flow Graph generation from source code. It defines a compact digraph-style textual CFG schema, a task-specific prompt, and a dataset derived from greengerong/leetcode (Java/Python/JavaScript) by Tree-sitter AST-to-CFG conversion plus four types of error augmentation that share the clean-code CFG labels. Models are fine-tuned for 3 epochs with Unsloth and evaluated with node/edge Precision, Recall, F1, EMR, and ZMR on clean vs. erroneous Java/Python splits and on held-out JavaScript. The authors conclude that fine-tuned lightweight models achieve strong node-level and moderate edge-level accuracy, remain robust to the injected errors, and exhibit partial cross-language generalization.

Significance. If the robustness and cross-language claims hold, the work would offer a practical, language-agnostic alternative to bytecode- and AST-based CFG tools for incomplete or syntactically broken code—precisely the setting where Soot, Spoon, and similar tools fail—while remaining deployable on modest hardware. Strengths include a multi-model comparison spanning code-specific and general-purpose lightweight LLMs, an explicit unified output schema, a held-out-language test, and public-style metrics (F1/EMR/ZMR) reported consistently. The contribution is incremental relative to prior LLM-CFG work (e.g., Huang et al.) but useful if the error-handling advantage can be substantiated against real baselines.

major comments (4)
  1. [§III-D, Q2, Figures 7–8] §III-D (Dataset Construction) and Q2 (Figures 7–8): The central claim that models work “particularly when the input code is incomplete or erroneous” (Abstract, §V) rests on the unvalidated premise that the four injected error types leave control-flow structure unchanged, so every erroneous variant re-uses the clean-code digraph label. No manual audit, re-parsing attempt, or reachability check is reported. If a non-negligible fraction of edits actually remove branches, alter statement order, or change reachability, then (a) a large portion of the 13 331 training targets is incorrect, (b) models are rewarded for ignoring real structural change, and (c) the near-zero clean–erroneous deltas become uninterpretable. This assumption is load-bearing for both the robustness narrative and the claimed advantage over parser-based generators; it needs either empirical validation on a sampled subset o
  2. [§IV, Related Work §II] §IV Evaluation: There is no quantitative comparison against traditional CFG generators (Soot, Spoon, Py2CFG, or the authors’ own Tree-sitter converter) on the clean subset, nor against zero-shot/few-shot large LLMs or the chain-of-thought pipeline of Huang et al. [6] that the related-work section itself cites. Without these baselines it is impossible to judge whether the reported Node F1 ≈ 0.95 / Edge F1 ≈ 0.65 constitutes an advance, a parity result, or a regression, or whether the models truly succeed where parsers fail. At minimum, success/failure rates of a conventional tool on the same clean and (where possible) erroneous test items should be reported.
  3. [§IV-B, Figures 5–6, 9] §IV-B/D and Figures 5–6, 9: Edge-level F1 remains substantially lower than node-level F1 across all models (best Edge F1 0.65–0.66 after ZMR removal; JavaScript Edge F1 ≈ 0.34–0.40). Because a CFG’s primary semantic content is the control-flow edges, the headline “promising results” overstate structural fidelity when edge recovery is only moderate and EMR is low. The paper should either elevate edge metrics as the primary success criterion or provide a qualitative analysis of the dominant edge-error modes (missing true branches, spurious edges, incorrect loop back-edges).
  4. [§III-D, §IV-A] §III-D / §IV-A: All ground-truth labels are produced by a single automatic AST-to-CFG pipeline run exclusively on clean code. Consequently the evaluation largely measures how well the fine-tuned models imitate that particular converter rather than independently correct CFG semantics. Combined with the shared-label error regime, this weakens claims of general CFG generation capability and of superiority on imperfect inputs. A small human-validated or multi-tool consensus subset would strengthen the evaluation.
minor comments (5)
  1. [Abstract, §I] Abstract and §I: “programming language not included in the fine-tuning data” is missing an article; several sentences contain doubled spaces or hyphenation artifacts (e.g., “syntax -error-free”, “unifi ed”).
  2. [§III-B] Figure 1–4 captions and body text refer to “Figure.1”, “Figure .3” inconsistently; the digraph example in Figure 4 is hard to parse without line breaks or syntax highlighting.
  3. [§IV-A] §IV-A: Exact train/test split sizes per language and the precise error-injection operators (e.g., which symbols are deleted, how incompleteness is realized) are not specified beyond high-level categories; reproducibility would benefit from a short appendix or release of the generation scripts.
  4. [§III-E, Figures 5–9] Model naming is inconsistent across text and figures (QwenCoder vs Qwen2.5-Coder, Qwen-4B vs Qwen3-4B, Phi-4B vs Phi-4-mini-3.8B). Align names with the Hugging Face identifiers used.
  5. [§II] Related-work discussion of Huang et al. [6] correctly notes cost limitations but does not clarify whether any of the same incomplete-code scenarios were re-used; a one-sentence positioning would help.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild construction-level circularity only in the robustness (erroneous-code) evaluation: shared CFG labels are assigned by design, so clean-vs-erroneous metric gaps partly reflect that protocol rather than independent verification.

  1. other [§III-D Dataset Construction (error augmentation); Q2 evaluation and Figures 7–8]
    "Importantly, each erroneous variant shares the same CFG label as its original clean sample because these minor errors are designed not to change the intended control-flow structure. This strategy simulates real-world development scenarios… By introducing such samples into the training data, the model is encouraged to learn the intended control-flow semantics…"

    The ground-truth CFGs used both for training and for the clean-versus-erroneous metric comparison are forced to be identical by the authors’ construction. Therefore any successful mapping of both clean and noisy inputs onto that common target necessarily produces small performance differences; the reported robustness (and the Abstract/§V claim that the approach works “particularly when the input code is incomplete or erroneous”) partly measures fidelity to this shared-label regime rather than an independently verified preservation of control flow under the injected faults.

full rationale

This is an empirical fine-tuning paper, not a first-principles derivation. The main results (node/edge F1, EMR/ZMR after supervised instruction tuning of six lightweight LLMs) are ordinary supervised-learning scores against automatically generated digraph labels; that is standard and not definitional circularity. The sole mild circularity arises in the error-augmentation protocol of §III-D and the Q2 comparison that rests on it: each of the four injected error types is deliberately paired with exactly the same CFG that Tree-sitter produced for the clean original, so the evaluation targets for clean and erroneous inputs are identical by construction. Consequently the near-zero clean–erroneous deltas reported in Figures 7–8 are partly forced by the shared-label design (any model that maps both inputs to the common target will look robust). The paper supplies no independent check that the injected faults truly leave control-flow structure unchanged. This does not collapse the overall claim that the models can emit the authors’ digraph format, nor does it involve self-citation, uniqueness theorems, or fitted parameters renamed as predictions; it is therefore scored only 2. No other load-bearing steps reduce to their inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on supervised fine-tuning against automatically generated digraph CFGs, the assumption that minor code errors preserve control-flow labels, and standard PEFT/instruction-tuning practice. No physical constants or fitted scientific parameters; free choices are engineering (epochs, model set, error types, digraph schema).

free parameters (3)
  • training_epochs
    Fixed to 3 for all models (§IV-A) without ablation; affects reported F1 and ZMR.
  • error_augmentation_types_and_count
    Four error variants per sample (missing symbols, typos, undefined variables, incomplete statements) chosen by authors (§III-D); defines the robustness experiment.
  • unified_digraph_schema_choices
    Node shapes (rectangle/diamond/hexagon/parallelogram) and compact digraph text format (§III-B) are design choices that define the prediction target and token budget.
assumptions (4)
  • domain assumption Tree-sitter AST traversal following common AST-to-CFG rules yields correct ground-truth CFGs for clean LeetCode solutions.
    §III-D uses this pipeline as the sole label source; no human validation or tool comparison is reported.
  • ad hoc to paper Injected minor errors do not change the intended control-flow structure, so clean and erroneous samples share the same CFG label.
    Stated explicitly in §III-D; load-bearing for the robustness claim in Q2.
  • domain assumption Supervised instruction fine-tuning of ~3–7B models is a valid way to induce structured CFG generation from source text.
    Standard LLM fine-tuning premise used throughout §III-E and §IV.
  • domain assumption Node/edge set matching (Precision/Recall/F1) plus EMR/ZMR adequately measures CFG quality for the stated claims.
    §IV-B defines metrics; exact matching rules for statement labels and edge identity are only sketched.
invented entities (1)
  • Unified digraph-style CFG textual schema (method_name, id/label/shape nodes, from->to edges)
    purpose: Standardize LLM outputs across tools/models and reduce token cost while preserving control-flow semantics.
    Introduced in §III-B as the training/evaluation target; not an external physical entity, but a paper-specific representation that defines success.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Finetuning Lightweight LLMs for Control Flow Graph Generation." pith.science (2026). https://pith.science/paper/UU6WFNTC

@misc{pith2026260704582,
  author       = {Pith},
  title        = {Pith review of: Finetuning Lightweight LLMs for Control Flow Graph Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UU6WFNTC}},
  note         = {Machine review of arXiv:2607.04582}
}
read the original abstract

Control Flow Graph (CFG) is an important program representations for software analysis, code understanding, and software maintenance. Traditional CFG generation techniques mainly rely on bytecode or abstract syntax trees. However, these approaches usually require complete, compilable, and syntax error-free code, which limits their applicability to incomplete or erroneous code. Furthermore, they often depend on language specific tools, making it difficult to support multiple programming languages in a unified manner. To address these limitations, this paper investigates the use of fine-tuned lightweight large language models (LLMs) for CFG generation. We first design a unified CFG output format and a task-specific fine-tuning prompt for CFG generation. Then, we construct a dataset based on an existing LeetCode dataset through automatic CFG generation and error augmentation. We evaluate the proposed approach on six lightweight LLM models, including three code-specific LLMs: CodeLlama, QwenCoder, and DeepSeekCoder; and three general purpose LLMs: Llama3.2-3B, Qwen-4B, and Phi-4B. The experimental results show that, through fine-tuning, lightweight LLMs achieve promising results for CFG generation, particularly when the input code is incomplete or erroneous. It also demonstrates cross-language generalization capability on programming language not included in the fine-tuning data.

Figures

Figures reproduced from arXiv: 2607.04582 by the authors.

Figure 2
Figure 2. Example code of CFG representation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Control flow graph of the example code [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 1
Figure 1. Unified CFG representation schema In this format, method_name denotes the name of the target method or function. Each id uniquely identifies a CFG node according to the original statement in the source code. The label field records the source-code statement represented by the node, while the shape field indicates the type of the statement. Specifically, rectangle is used for ordinary process statements, diamond for … view at source ↗
Figures from the paper (4 more)
Figure 7
Figure 7. Figure 7: The Difference of overall performance [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]
Figure 5
Figure 5. Figure 5: Overall performance on test dataset [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: ZMR Samples Removed performance Furthermore, we removed the zero-matched samples, where the predictions contained no valid matched nodes or edges, and recalculated the performance. As shown in Figure.6, QwenCoder still achieved the best overall performance, with a Node…
Figure 9
Figure 9. Figure 9: The overall performance on JavaScript dataset V. CONCLUSION In this paper, we explored the use of fine-tuned lightweight large language models for CFG generation. We first designed a unified CFG representation and a task-specific prompting strategy to standardize CFG g…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [6]

    Spoon: A library for implementing analyses and transformations of Java source code,

    R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A library for implementing analyses and transformations of Java source code,” Softw. Pract. Exp., vol. 46, no. 9, pp. 1155–1179, 2016

    2016

  2. [1]

    You are a control flow graph generator. Task: Generate the Control Flow Graph (CFG) for the given method

    for Python, both of which construct CFGs from parsed syntax trees. These tools have significantly advanced CFG construction by improving automation and supporting multiple programming languages. However, existing CFG generation approaches still suffer from several practical limitations. For bytecode -based approaches, source code must be complete and comp...

    work page doi:10.18293/seke2026-036

  3. [2]

    Control flow analysis,

    F. E. Allen, “Control flow analysis,” SIGPLAN Not., vol. 5, no. 7, pp. 1– 19, 1970

    1970

  4. [3]

    Constructing more complete control flow graphs utilizing directed gray-box fuzzing,

    K. Zhu, Y. Lu, H. Huang, L. Yu, and J. Zhao, “Constructing more complete control flow graphs utilizing directed gray-box fuzzing,” Appl. Sci., vol. 11, no. 3, Art. no. 1351, 2021

    2021

  5. [4]

    WALA: Static analysis framework for Java

    IBM, “WALA: Static analysis framework for Java.” [Online]. Available: https://sourceforge.net/projects/wala/ [Accessed: Jan. 5, 2026]

    2026

  6. [5]

    Soot: A Java bytecode optimization framework,

    R. Vallée-Rai, P. Co, E. Gagnon, L. Hendren, P. Lam, and V. Sundaresan, “Soot: A Java bytecode optimization framework,” in CASCON First Decade High Impact Papers, 2010, pp. 214–224

    2010

  7. [7]

    AI chain on large language model for unsupervised control flow graph generation for statically-typed partial code,

    Q. Huang, Z. Zou, Z. Xing, Z. Zuo, X. Xu, and Q. Lu, “AI chain on large language model for unsupervised control flow graph generation for statically-typed partial code,” arXiv preprint arXiv:2306.00757, 2023

    arXiv 2023

  8. [8]

    A control flow graph generation method for Java projects,

    H. Tran and P. Hung, “A control flow graph generation method for Java projects,” VNU J. Sci.: Comput. Sci. Commun. Eng., vol. 40, 2024

    2024

Show all 32 references
  1. [9]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol., 2024

    2024

  2. [10]

    Survey of hallucination in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, pp. 1–38, 2023

    2023

  3. [11]

    Faithful reasoning using large language models,

    A. Creswell and M. Shanahan, “Faithful reasoning using large language models,” arXiv preprint arXiv:2208.14271, 2022

    2022 arXiv

  4. [12]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  5. [13]

    GraphCodeBERT: Pre -training code representations with data flow,

    D. Guo, S. Ren, S. Lu, F. Zhang, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu, et al., “GraphCodeBERT: Pre -training code representations with data flow,” arXiv preprint arXiv:2009.08366, 2020

    2009 arXiv

  6. [14]

    Capturing source code semantics via tree- based convolution over API-enhanced AST,

    L. Chen, W. Ye, and S. Zhang, “Capturing source code semantics via tree- based convolution over API-enhanced AST,” in Proc. 16th ACM Int. Conf. Comput. Frontiers, 2019, pp. 174–182

    2019

  7. [15]

    Detecting code clones with graph neural network and flow-augmented abstract syntax tree,

    W. Wang, G. Li, B. Ma, X. Xia, and Z. Jin, “Detecting code clones with graph neural network and flow-augmented abstract syntax tree,” in Proc. IEEE 27th Int. Conf. Softw. Anal., Evol. Reeng. (SANER), 2020, pp. 261–271

    2020

  8. [16]

    Deep code comment generation,

    X. Hu, G. Li, X. Xia, D. Lo, and Z. Jin, “Deep code comment generation,” in Proc. 26th Conf. Program Comprehension, 2018, pp. 200–210

    2018

  9. [17]

    Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,

    H. Wei and M. Li, “Supervised deep features for software functional clone detection by exploiting lexical and syntactical information in source code,” in Proc. 26th Int. Joint Conf. Artif. Intell. (IJCAI), 2017, pp. 3034–3040

    2017

  10. [18]

    ModularTree network for source code representation learning,

    W. Wang, G. Li, S. Shen, X. Xia, and Z. Jin, “ModularTree network for source code representation learning,” ACM Trans. Softw. Eng. Methodol., vol. 29, pp. 1–23, 2020

    2020

  11. [19]

    A novel neural source code representation based on abstract syntax tree,

    J. Zhang, X. Wang, H. Zhang, H. Sun, K. Wang, and X. Liu, “A novel neural source code representation based on abstract syntax tree,” in Proc. IEEE/ACM 41st Int. Conf. Softw. Eng. (ICSE), 2019, pp. 783–794

    2019

  12. [20]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pondéde, O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  13. [21]

    Code Llama: Open foundation models for code,

    B. Rozière et al., “Code Llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  14. [22]

    Qwen2.5 -Coder technical report,

    B. Hui et al., “Qwen2.5 -Coder technical report,” arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  15. [23]

    DeepSeek -Coder: When the large language model meets programming— The rise of code intelligence,

    D. Guo et al., “DeepSeek -Coder: When the large language model meets programming— The rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  16. [24]

    Graphviz—Open source graph drawing tools,

    J. Ellson, E. R. Gansner, E. Koutsofios, S. C. North, and G. Woodhull, “Graphviz—Open source graph drawing tools,” in Graph Drawing. Berlin, Germany: Springer, 2001, pp. 483–484

    2001

  17. [25]

    Available: https://py2cfg.readthedocs.io/ [Accessed: Jan

    Py2CFG [Online]. Available: https://py2cfg.readthedocs.io/ [Accessed: Jan. 5, 2026]

    2026

  18. [26]

    Claude Code

    Anthropic, “Claude Code.” [Online]. Available: https://www.anthropic.com/product/claude-code [Accessed: Jan. 5, 2026]

    2026

  19. [27]

    greengerong/leetcode dataset

    Hugging Face, “greengerong/leetcode dataset.” [Online]. Available: https://huggingface.co/datasets/greengerong/leetcode [Accessed: Jan. 5, 2026]

    2026

  20. [28]

    Available: https://tree -sitter.github.io/](https://tree- sitter.github.io/

    Tree-sitter, [Online]. Available: https://tree -sitter.github.io/](https://tree- sitter.github.io/. [Accessed: Jan. 5, 2026]

    2026

  21. [29]

    [Online]

    Unsloth. [Online]. Available: https://github.com/unslothai [Accessed: Jan. 5, 2026]

    2026

  22. [30]

    Llama 3.2

    Meta, “Llama 3.2.”. [Online]. Available: https://github.com/meta - llama/llama-models/blob/main/models/llama3_2 [Accessed: Jan. 5, 2026]

    2026

  23. [31]

    Phi -4-mini-instruct

    Microsoft, “Phi -4-mini-instruct.”. [Online]. Available: https://huggingface.co/microsoft/Phi-4-mini-instruct [Accessed: Jan. 5, 2026]

    2026

  24. [32]

    Qwen3 -4B

    Qwen, “Qwen3 -4B.”. [Online]. Available: https://huggingface.co/Qwen/Qwen3-4B [Accessed: Jan. 5, 2026].. 0.69 0.34 0.15 0.26 0.72 0.38 0.22 0.07 0.73 0.40 0.14 0.47 0.69 0.38 0.18 0.06 0.70 0.37 0.20 0.06 0.70 0.36 0.17 0.17 0.00 0.20 0.40 0.60 0.80 Node F1 Edge F1 Node-EMR No...

    2026

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.