Pith. sign in

REVIEW 3 major objections 6 minor 24 references

CodableLLM: Automating Decompiled and Source Code Mapping for LLM Dataset Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper presents CodableLLM, an open-source Python framework that automates the end-to-end mapping of decompiled binary functions to their corresponding source-code functions, so that large aligned datasets for training code-focused…

desk verdict Useful, genuinely open-source tool for building source/decompiled datasets, but the central mapping-correctness claim is unvalidated and needs work before acceptance. read the letter →

arxiv 2507.22066 v1 pith:C7MUMNJG submitted 2025-07-02 cs.SE cs.CR

Dylan Manuel , Paul Rad This is my paper
classification cs.SEcs.CR
keywords largelanguagemodelsautomationreverseengineeringsoftwaresecuritydatasetgenerationdecompiledcodesource-codemappingfunctionalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CodableLLM is a Python framework whose stated purpose is to close a gap in code dataset generation: existing source-code corpora do not contain decompiled binaries, and existing binary-analysis tools do not produce aligned source-to-decompiled pairs, so there has been no scalable, automated way to build the aligned datasets needed for training LLMs on decompilation and reverse engineering. The paper claims that CodableLLM is the first open-source, configurable, language-agnostic framework to automate source extraction, decompilation, symbol-based mapping, and dataset export as one end-to-end pipeline. If that claim holds, the practical consequence is that aligned binary/source datasets, which have previously required manual or ad hoc construction, can be generated routinely from any buildable repository, making research on binary function recovery, binary similarity, and AI-driven decompilation far more reproducible. The evaluation on the libhv C library reports that the parallel pipeline cuts decompilation time from 514.52 seconds to 55.93 seconds, while also noting that a naive single-threaded run produced slightly more successful function mappings because of concurrency errors in Ghidra.

What carries the argument

The central object is the CodableLLM multi-stage pipeline itself: a repository manager clones and builds the project; Tree-Sitter-based extractors traverse ASTs to pull out source functions; a Ghidra-based decompiler turns binaries into pseudo-C and extracts decompiled functions; a mapper aligns the two function sets by comparing symbol names, with configurable heuristics such as filename and namespace consistency; and pandas DataFrames export the aligned pairs to CSV, JSON, or other structured formats. The load-bearing mechanism is the symbol-name correspondence between a decompiled function and a source function, because every dataset row depends on that alignment. Parallelism, orchestrated through Prefect tasks, is what makes the pipeline scalable and is the mechanism behind the reported speedups.

What would settle it

Take a representative open-source C project, build it, strip the symbol table from the produced binaries, run CodableLLM, and count how many decompiled functions are correctly paired with their source functions; if the correctly aligned count collapses to near zero while the pipeline still emits a dataset, the symbol-based mapping is the sole mechanism and the framework does not deliver on its stated goal for stripped binaries. A complementary check is to compile two translation units that define different functions with the same name and verify whether the mapper pairs each decompiled function with the wrong source body.

Watch

Extended reading notes

Core claim

The central claim is that CodableLLM provides the first open-source infrastructure that fully automates the mapping between decompiled functions and their corresponding source functions for LLM dataset generation. The pipeline takes a local or remote repository, builds it, extracts source functions with Tree-Sitter, decompiles the resulting binaries with Ghidra, aligns the two representations by matching symbol names, and exports the aligned pairs as CSV or JSON. The paper demonstrates the framework on libhv, reporting a roughly tenfold reduction in decompilation time under parallelism and sub-second mapping and export times; it also reports that the parallel execution produced slightly fewer successful mappings than the naive run, which the authors attribute to Ghidra's concurrency-related errors. These results are offered as evidence that CodableLLM is both fast and practical for real-world dataset construction, while the extensibility of extractors, decompilers, and mapping strategies supports the claim of language-agnostic, configurable automation.

Load-bearing premise

The entire dataset-generation pipeline depends on function names surviving from the source code into the decompiled binary; when symbols are stripped or renamed, the mapper has nothing to match, direct alignment becomes impossible, and the pipeline's central promise fails for those inputs, a limitation the paper itself acknowledges.

Editorial extensions

If this is right

  • Aligned binary/source datasets can be generated from any buildable repository without hand-written scripts, enabling reproducible benchmarks for function recovery and binary analysis.
  • LLMs trained on such paired data could learn to translate decompiled pseudo-C back into high-level source, potentially improving automated decompilation and downstream vulnerability analysis.
  • Because users can plug in custom extractors, decompilers, build commands, and mapping heuristics, the same pipeline can be extended beyond C and Ghidra to other languages and toolchains.
  • The reported speedup makes repository-scale dataset construction practical, not just small curated corpora, which lowers the cost of scaling training data for code-focused models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves implicit: symbol-name mapping could be fused with embedding-based or structural similarity matching, which would let the same pipeline handle stripped binaries instead of merely excluding them.
  • Dataset quality, not just throughput, will become the limiting factor if CodableLLM is widely adopted; the paper's own observation that parallel runs produce slightly fewer successful mappings points to the need for a validation or quality-filter stage before training.
  • The same pipeline could be applied in reverse to existing source-only corpora, generating binary views of already-curated code and effectively creating pseudo-parallel code corpora for binary-focused LLM training without new repository collection.
  • For security applications the paper mentions, such as malware analysis, symbol-based alignment will rarely be available in the wild, so the framework's practical security value depends on the better heuristics the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents CodableLLM, a Python framework that automates construction of datasets linking source-code functions to decompiled functions from binaries. The pipeline clones and builds a repository, extracts functions from source using Tree-Sitter, decompiles selected binaries with Ghidra, maps source to decompiled functions via symbol-name heuristics, and exports CSV/JSON datasets. The evaluation is primarily a runtime comparison against a single-threaded pipeline, run on the libhv C library and, in a summary table, on several other repositories. The paper claims to be the first open-source framework for automated source-to-decompiled mapping for LLM datasets.

Significance. If the mapping quality were validated, CodableLLM would be a useful infrastructure contribution: it is open-source, packaged on PyPI, documented, and ships a dataset. The runtime speedups (roughly 10x for decompilation on libhv) are supported by measurements. However, the central quality claim rests on an unvalidated symbol-name matching heuristic; no precision or recall is measured. The value of the framework for LLM dataset generation therefore remains unestablished, and the paper's title claim of 'high-quality datasets' is not evidenced.

major comments (3)
  1. [IV-C, VI] The conclusion in Section VI that CodableLLM is 'capable of generating high-quality datasets' is not supported by the evaluation in Section IV. Algorithm 1 relies on mapper.is_potential_match, which compares symbol names, but the paper reports no precision, recall, or manual verification of the emitted decompiled_uid-to-source_uid pairs. In unstripped binaries, name collisions among static helpers, C++ overloading, compiler name mangling, and decompiler-generated names (e.g., FUN_*) can cause silent misalignment. The runtime figures in Table III cannot substitute for a correctness measurement. Please add a ground-truth evaluation (e.g., a labeled subset, debug-symbol-based oracle, or downstream function-recovery benchmark) and report match accuracy and error cases.
  2. [V-B, Table III] Section V-B states that the naive single-threaded run produced 'slightly more successful function mappings' than the parallel runs, but no mapping counts are reported anywhere in the paper. Table III lists only execution times. Without the number of extracted functions, matched pairs, and mapping rate per workflow and per project, the claimed 'practical effectiveness' in Section I cannot be assessed. Please report these figures, including the drop rate for stripped runs.
  3. [II-D] The claim that 'to date, no open-source framework provides a configurable, extensible, and language-agnostic infrastructure' for this mapping is a strong novelty claim that the paper supports only with a feature checklist in Table I. A more direct comparison with existing artifact pipelines, including Assemblage and SourceFinder cited in the same section, is needed to substantiate the 'first' claim. At minimum, state explicitly what reverse-mapping capability each alternative lacks and any overlap with CodableLLM's mapping heuristic.
minor comments (6)
  1. [Abstract, Section I] There is a typo 'LLMS' that should be 'LLMs', and the expansion 'COdeDAtaset BuiLdEr forLLMs' is inconsistent with the name CodableLLM used elsewhere.
  2. [IV-C] The phrase 'simulated single-threaded extraction and decompilation pipeline' is ambiguous; clarify whether this is a separate implementation or the same pipeline run with concurrency disabled.
  3. [Algorithm 1] The mapping loop (lines 27-33) is O(N×M) in the number of functions; for the 'at scale' claim in Section II-D, discuss indexing or complexity, or reduce before claiming large-scale suitability.
  4. [Table III] The column headers 'Concurrent', 'Parallel', and 'Concurrent + Strip' are not defined; also clarify why stripped runs are only reported for libhv and Tokio, and what 'Strip' stands for (symbol stripping?).
  5. [V-B] The claim that Ghidra 'may produce inconsistent results in multi-threaded environments' is plausible but unsupported; include error counts or specific failures observed.
  6. [References] Several references are incomplete or have formatting artifacts (e.g., [4] author name, [10] venue, and 'V otipka' spacing); please polish the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CodableLLM's pipeline is a tool description plus a wall-clock performance comparison; no prediction is derived from fitted inputs or from load-bearing self-citations.

full rationale

The paper describes an automated dataset-generation pipeline (build, source extraction, decompilation, symbol-name mapping, export) and evaluates it by measuring wall-clock times against a naive single-threaded baseline. There is no derived quantity that is defined in terms of the result it allegedly predicts, and no fitted parameter is relabeled as a prediction. The symbol-based mapping is a heuristic design choice whose accuracy is not validated against ground truth; that is an evidentiary gap, not circularity. The novelty claim of being the first open-source framework is not established by any circular derivation, and all cited prior work is external to the authors. The limitation in Section V-B that stripped binaries make direct alignment impossible is an honest stated restriction, not a circular step. The lack of an external benchmark is a validation weakness but does not reduce the paper's claims to their inputs. Therefore no enumerated circularity pattern is present, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the paper contains no numerical model. The framework relies on two domain assumptions: that symbol-name matching suffices for alignment, and that Ghidra's function decomposition matches source-level functions. No invented entities are introduced.

assumptions (3)
  • domain assumption Decompiled functions can be reliably aligned to source functions by matching symbol names.
    Section III-D uses symbol-based matching as the default mapper; Section V-B admits this fails for stripped binaries.
  • domain assumption Ghidra's decompilation output contains parseable function boundaries that correspond to source-level functions.
    Section III-C builds on Ghidra's pseudo-C output for function-level extraction without validating granularity equivalence.
  • domain assumption The timing comparison to a simulated single-threaded pipeline is a fair baseline for demonstrating efficiency.
    Section IV-C compares against a naive single-extractor and single-decompiler simulation, not an external tool baseline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodableLLM: Automating Decompiled and Source Code Mapping for LLM Dataset Generation." pith.science (2026). https://pith.science/paper/C7MUMNJG

@misc{pith2026250722066,
  author       = {Pith},
  title        = {Pith review of: CodableLLM: Automating Decompiled and Source Code Mapping for LLM Dataset Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7MUMNJG}},
  note         = {Machine review of arXiv:2507.22066}
}
read the original abstract

The generation of large, high-quality datasets for code understanding and generation remains a significant challenge, particularly when aligning decompiled binaries with their original source code. To address this, we present CodableLLM, a Python framework designed to automate the creation and curation of datasets by mapping decompiled functions to their corresponding source functions. This process enhances the alignment between decompiled and source code representations, facilitating the development of large language models (LLMs) capable of understanding and generating code across multiple abstraction levels. CodableLLM supports multiple programming languages and integrates with existing decompilers and parsers to streamline dataset generation. This paper presents the design and implementation of CodableLLM, evaluates its performance in dataset creation, and compares it to existing tools in the field. The results demonstrate that CodableLLM offers a robust and efficient solution for generating datasets tailored for code-focused LLMS.

Figures

Figures reproduced from arXiv: 2507.22066 by the authors.

Figure 1
Figure 1. The main workflow of CodableLLM. In this example, an unbuilt local repository, demo-c-repo, is used as input to CodableLLM, specifying that the make command will be used to build the repository. The resulting binaries, main_app and tool, are built from this process. CodableLLM first executes the build command to produce these binaries. It then extracts all source code functions from the repository’s source files. Ne… view at source ↗
Figure 2
Figure 2. Layered architecture of CodableLLM, illustrating the configuration interface, high-level orchestration, and low-level Prefect-parallelized extraction and decompilation with support for user-defined extensions. Field Description decompiled_uid A unique identifier assigned to each decompiled function entry. assembly The disassembled assembly instructions corresponding to the decompiled function, extracted from the bin… view at source ↗
Figure 3
Figure 3. Execution time (in minutes) for function extraction and decompilation across multiple open-source repositories. Concurrent times were measured [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

  1. [1]

    The adverse effects of code duplication in machine learning models of code,

    M. Allamanis, “The adverse effects of code duplication in machine learning models of code,” inProceedings of the ACM SIGPLAN Inter- national Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! ’19). ACM, 2019, pp. 143–153

    work page 2019

  2. [2]

    Quality in, quality out: Investigating training data’s role in ai code generation,

    C. Improta, R. Tufano, P. Liguori, D. Cotroneo, and G. Bavota, “Quality in, quality out: Investigating training data’s role in ai code generation,” arXiv preprint arXiv:2503.11402, 2025

    arXiv 2025

  3. [3]

    A plea for utilising synthetic data when performing machine learning based cyber-security experiments,

    S. Abt and H. Baier, “A plea for utilising synthetic data when performing machine learning based cyber-security experiments,” inProceedings of the 2014 ACM Workshop on Artificial Intelligent and Security (AISec). ACM, 2014, pp. 37–45

    work page 2014

  4. [4]

    tree-sitter/tree-sitter: An incremental parsing system for programming tools,

    M. Smolens, “tree-sitter/tree-sitter: An incremental parsing system for programming tools,” GitHub repository, 2019. [Online]. Available: https://github.com/tree-sitter/tree-sitter

    work page 2019

  5. [5]

    srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration,

    J. I. Maletic, M. L. Collard, and A. Marcus, “srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration,” inProceedings of the 29th IEEE International Conference on Software Maintenance (ICSM), 2013. [Online]. Available: https://www.cs.kent.edu/∼jmaletic/papers/ICSM13-srcML.pdf

    work page 2013

  6. [6]

    Codesearchnet challenge: Evaluating the state of semantic code search,

    H. Husainet al., “Codesearchnet challenge: Evaluating the state of semantic code search,”arXiv preprint arXiv:1909.09436, 2019. [Online]. Available: https://arxiv.org/abs/1909.09436

    arXiv 1909

  7. [7]

    The pile: An 800gb dataset of diverse text for language modeling,

    L. Gaoet al., “The pile: An 800gb dataset of diverse text for language modeling,”arXiv preprint arXiv:2101.00027, 2020. [Online]. Available: https://arxiv.org/abs/2101.00027

    arXiv 2020

  8. [8]

    The stack: A 6.4 tb dataset of permissively licensed source code,

    “The stack: A 6.4 tb dataset of permissively licensed source code,” BigCode Project, 2021. [Online]. Available: https://www. bigcode-project.org/docs/about/the-stack/

    work page 2021

Show all 24 references
  1. [9]

    Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks,

    R. Puri, D. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, V . Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss, “Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks...

    2021 arXiv

  2. [10]

    A survey of machine learning methods and challenges for windows malware classification,

    E. Raff and C. Nicholas, “A survey of machine learning methods and challenges for windows malware classification,” inNeurIPS 2020 Workshop on ML Retrospectives, Surveys, & Meta-Analyses (ML-RSA), 2020, arXiv:2006.09271

    2020 arXiv

  3. [11]

    An investigation of online reverse engineering community discussions of ghidra,

    D. V otipkaet al., “An investigation of online reverse engineering community discussions of ghidra,” inProceedings of the 2021 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), 2021. [Online]. Available: https://www.eecs.tufts.edu/ ∼dvotipka/files/papers/V ...

    2021

  4. [12]

    Ida pro: The interactive disassembler,

    “Ida pro: The interactive disassembler,” Hex-Rays SA. [Online]. Available: https://hex-rays.com/ida-pro

  5. [13]

    Xda: Accurate, robust disassembly with transfer learning,

    K. Pei, B. Pak, J. Jang, and S. Jana, “Xda: Accurate, robust disassembly with transfer learning,” inProceedings of the Network and Distributed System Security Symposium (NDSS), 2021

    2021

  6. [14]

    Llm4decompile: Decom- piling binary code with large language models,

    H. Tan, Q. Luo, J. Li, and Y . Zhang, “Llm4decompile: Decom- piling binary code with large language models,”arXiv preprint arXiv:2403.05286, 2024

    2024 arXiv

  7. [15]

    Extending source code pre-trained language models to summarise decompiled binaries,

    A. Al-Kaswan, T. Ahmed, M. Izadi, A. A. Sawant, P. Devanbu, and A. van Deursen, “Extending source code pre-trained language models to summarise decompiled binaries,” inProceedings of the 30th IEEE International Conference on Software Analysis, Evolution and Reengi- neering (SA...

    2023

  8. [16]

    Assemblage: Automatic binary dataset construction for machine learning,

    C. Liu, R. Saul, Y . Sun, E. Raff, M. Fuchs, T. S. Pantano, J. Holt, and K. Micinski, “Assemblage: Automatic binary dataset construction for machine learning,” inAdvances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024

    2024

  9. [17]

    Sourcefinder: Finding malware source-code from publicly available repositories in github,

    M. O. F. Rokon, R. Islam, A. Darki, E. E. Papalexakis, and M. Faloutsos, “Sourcefinder: Finding malware source-code from publicly available repositories in github,” inProceedings of the 23rd International Sympo- sium on Research in Attacks, Intrusions and Defenses (RAID). USEN...

    2020

  10. [18]

    pandas: a foundational python library for data analysis and statistics,

    W. McKinney, “pandas: a foundational python library for data analysis and statistics,” 2010. [Online]. Available: https://www.researchgate.net/ publication/265194455 pandas a Foundational

    2010

  11. [19]

    Prefect: The modern workflow orchestration framework,

    I. Prefect Technologies, “Prefect: The modern workflow orchestration framework,” https://www.prefect.io, 2024, accessed: 2024-03-31

    2024

  12. [20]

    libhv: A cross-platform c network library,

    W. He, “libhv: A cross-platform c network library,” GitHub repository,

  13. [21]

    Binary ninja: Reverse engineering platform,

    “Binary ninja: Reverse engineering platform,” Vector35, 2024. [Online]. Available: https://binary.ninja

    2024

  14. [22]

    Radare2: Open-source reverse engineering framework,

    “Radare2: Open-source reverse engineering framework,” Radare Project,

  15. [24]

    Available: https://rada.re/n

    [Online]. Available: https://rada.re/n

  16. [2024]

    Available: https://github.com/ithewei/libhv

    [Online]. Available: https://github.com/ithewei/libhv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.