Pith. sign in

REVIEW 3 major objections 3 minor 5 references

This paper argues that fair benchmarking of LLMs on reverse engineering is limited by the reliability of binary-to-source alignment, not model capability, and that unpaired optimization comparisons overstate performance decay through surviv

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:09 UTC pith:QTJ5ZJOY

load-bearing objection A solid, well-scoped methodology paper — the eight-gate funnel and paired stable-key analysis are genuinely new, but the paired claim rests on a key that is asserted rather than independently validated. the 3 major comments →

arxiv 2607.07738 v2 pith:QTJ5ZJOY submitted 2026-07-07 cs.SE cs.AIcs.CRcs.PL

REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

Nicolas Koller , Andreas U. Schmidt This is my paper
classification cs.SE cs.AIcs.CRcs.PL
keywords reverse engineeringlarge language modelsbenchmarkingdecompilationfunction namingsurvivorship biasDWARFground truth alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Reforge, a provenance-tracked pipeline for building function-level ground truth in LLM reverse-engineering benchmarks. Its central claim is that the bottleneck is not the model but the reliability of mapping compiled binary functions back to their source declarations. High-confidence ground-truth yield falls from 87.2% to 65.9% across compiler optimization levels. Comparing aggregate means across levels compares different populations of functions; the apparent decay is mostly survivorship bias, not functions becoming harder to name. Paired analysis over a stable source-anchored key removes this confound.

Core claim

Reforge constructs ground truth by compiling synthetic C programs, extracting function inventories from DWARF and syntax trees, aligning them via compiler line directives, decompiling stripped binaries, and gating alignment quality through an eight-check confidence funnel. On a 15-program micro-benchmark, the funnel's high-confidence yield drops from 87.2% to 65.9% with optimization, while the population of evaluable functions shrinks from 220 to 132. When the same functions are tracked across optimization levels with a source-anchored key, per-function naming accuracy changes only marginally (−0.006 to +0.033 Token F1 across models), so the aggregate decline is attributable to which functio

What carries the argument

The stable, source-anchored function key: test case, declaration file, declaration line and column, and normalized function name. It tracks a logical function across optimization levels when binary-side identifiers change for roughly 86% of functions between the two lowest levels. Paired comparison over this key turns aggregate optimization comparisons into within-function comparisons. The eight-gate confidence funnel operationalizes alignment uncertainty as explicit, stratified tiers (GOLD, SILVER, BRONZE) without silently discarding functions that fail quality checks.

Load-bearing premise

The stable key assumes the source declaration line and column uniquely identify the same logical function across optimization levels; if inlining or merging changes the declaration-to-binary mapping, the paired analysis would pair different functions and the survivorship conclusion would be an artifact.

What would settle it

A manual audit of a random sample of paired functions checking whether the source-anchored key maps to the same logical function across levels; if mismatches are frequent, the survivorship-bias conclusion collapses. Alternatively, a comparison on a corpus with independently verified declaration sites that shows within-function decay reappearing once pairing is corrected would falsify the claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmarks claiming optimization-level effects should report alignment yield and per-stage attrition alongside accuracy, and use paired analysis over a source-anchored key.
  • Accuracy numbers reported without disclosing ground-truth reliability should be understood as describing a survivorship-filtered population, especially at higher optimization levels.
  • Ranked shortlists of candidate names (best of three) offer a practical boost in analyst-assistance settings, but intra-function heuristics cannot recover the oracle uplift because stripped functions retain a median of only two meaningful identifier tokens.
  • The methodological conclusions—attrition, shifting failure modes, and survivorship bias—are properties of the compilation-and-alignment process and are expected to hold for any toolchain emitting DWARF-equivalent metadata.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The survivorship-bias critique generalizes beyond function naming to any binary-analysis benchmark where compilation attrition selects simpler functions; the same paired-key logic applies to type recovery, summarization, and vulnerability detection evaluations.
  • The finding that roughly one in seven lexically zero-scoring predictions is semantically apt suggests current exact-match metrics undervalue synonym recovery; a standardized semantic metric would give fairer capability comparisons across models.
  • The stable-key design depends on the source declaration site surviving inlining and merging; a testable threat is whether the same declaration line can host different logical functions after transformation, which would mispair the analysis.
  • The near-zero uplift from corpus-derived re-ranking hints that useful signals for shortlist selection must come from beyond the function body, such as call-graph context or cross-reference frequency.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The manuscript argues that the principal bottleneck in benchmarking LLM-assisted reverse engineering is not model capability but the reliability of binary-to-source alignment under compiler optimization. It introduces Reforge, a provenance-tracked pipeline that constructs function-level ground truth from synthetic C programs compiled at four optimization levels, using a DWARF oracle and a Tree-sitter syntactic oracle, aligning them via line directives, decompiling stripped binaries with Ghidra, and scoring LLM predictions against source identifiers. Alignment uncertainty is operationalized as an eight-gate confidence funnel with GOLD/SILVER/BRONZE tiers, and function identity across optimization levels is tracked with a stable source-anchored key (test case + declaration file/line/column + normalized name). On a 15-program micro-corpus, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and the paper reports that unpaired optimization-level comparisons overstate performance decay through survivorship bias: per-function paired deltas across models range from −0.006 to +0.033, with Wilcoxon signed-rank p-values ≥ 0.209. Seven LLMs are evaluated on function naming, with modest top-1 accuracy, a small upper-bound benefit from top-K shortlists, and a semantic-similarity check bounding the lexical metric's underestimate.

Significance. The contribution is primarily methodological. If the findings hold, the paper provides a valuable corrective to common benchmark practice: reporting alignment yield and per-stage attrition, stratifying by confidence, and preferring paired analysis over a stable source-anchored key when comparing across optimization levels. Strengths include a deterministic ground-truth core verified by repeated hash comparison, a direct check of Ghidra decompiler determinism, an auditable attrition trail in which no function is silently discarded, and an openly available pipeline and evaluation notebooks. The paper also carefully labels the empirical LLM results as proof-of-concept, not general capability estimates. However, the central survivorship claim rests on the validity of the stable key, which is not independently validated; this needs to be addressed before the methodological conclusion can be considered fully supported.

major comments (3)
  1. [§3.4] The paired-analysis conclusion in §4.2 depends entirely on the stable key (test case + declaration file/line/column + normalized name) correctly identifying the same logical function at -O0 and -O3. The paper does not validate this key against any independent ground truth, such as manual inspection of a sample of functions or a compiler-verified mapping. Under inlining and merging, the address-overlap join can attach a decompiled binary function to one source declaration while the model is scored against a different or superposed function. If such mispairing is not rare, the reported paired deltas of −0.006 to +0.033 and Wilcoxon p ≥ 0.209 are what noise would look like, and the null result would not support the survivorship-bias explanation. A direct fix is to re-run the naming evaluation on the same corpus compiled with -O3 -fno-inline, where each source function survives as a separate
  2. [§4.2] The paired cohort is described only as 'functions evaluable at both the lowest and highest levels.' The paper does not report the cohort size, its distribution across GOLD/SILVER/BRONZE tiers, or whether the cohort includes weak-join and gated-out functions. This matters because the mispairing risk is highest for SILVER and BRONZE functions, whose address joins may be weak or ambiguous. The authors should report the number of paired functions, their tier composition, and the paired analysis separately for GOLD-only versus all tiers. A small stratum with weak joins could be driving the null result even if GOLD functions show a real within-function decay.
  3. [§5] The abstract states without qualification that 'unpaired optimization-level comparisons overstate performance decay through survivorship bias,' and §5 asserts this is 'expected to hold for any toolchain that emits DWARF-equivalent metadata.' The empirical support is a 15-program synthetic corpus compiled with a single compiler (GCC) for a single architecture (x86-64) and decompiled with Ghidra. The paper later lists cross-toolchain replication as future work, which is in tension with the unqualified claim. The authors should either temper the claim to the observed setting (GCC/x86-64/Ghidra) or provide at least one additional toolchain/architecture replication to justify the generalization.
minor comments (3)
  1. [§4.3] Section numbering error: '3.5 Tier Strata Revisited' appears after §4.3 and before §4.4; it should be renumbered (e.g., §4.4) or absorbed into §3.5 with cross-references adjusted.
  2. [Table 1] Gate 5 ('Strong address join (≥ 0.9)') and Gate 4 ('Overlap ratio ≥ 0.95') use thresholds that are presented as fixed. The paper lists a threshold sensitivity analysis as future work, but a brief statement of how sensitive the yield numbers and paired-delta results are to these thresholds would strengthen the reader's confidence that the findings are not artifacts of the hand-chosen values.
  3. [§4.4] The top-K oracle metric is correctly labeled an upper bound, but the sentence 'the best candidate appears at rank two or three rather than rank one, indicating a ranking or calibration gap' should be careful: the oracle selects the best candidate after seeing ground truth, so the rank distribution of improvements is not necessarily evidence of a realizable ranking gap. A minor rewording would avoid over-interpretation.

Circularity Check

0 steps flagged

No significant circularity: the survivorship-bias claim is an empirical paired-vs-unpaired comparison, not a construct defined into existence.

full rationale

The paper's central claim—that unpaired optimization-level comparisons overstate performance decay through survivorship bias—is an empirical statistical decomposition. It contrasts aggregate means across optimization levels with per-function paired deltas computed over a stable source-anchored key (Section 3.4). The paired deltas and Wilcoxon results are measured outcomes, not consequences of how the key is defined. The key (test case, declaration file/line/column, normalized name) does not encode the naming score, so the finding is not self-definitional. The confidence-funnel thresholds are design choices; no parameter is fitted to produce the survivorship result. The top-K oracle metric is explicitly labeled an upper bound, not a prediction. No self-citations appear in the reference list; the EUBA and decompiler-fidelity works cited are external and used as background or design motivation, not to forbid alternatives. The unvalidated stable key is a validity threat—mispairing under inlining/merging could make the paired null result an artifact—but a threat to validity is not circularity: if the key is wrong, the conclusion is incorrect, not tautological. No step reduces to its own inputs by construction, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim depends on the ground-truth oracle (DWARF and Tree-sitter) being correct, on the line-directive alignment, and on the stable key preserving function identity across optimization levels. The gate thresholds and K are hand-chosen design parameters.

free parameters (3)
  • GOLD overlap gate threshold = 0.95
    Gate 4 requires overlap ratio >= 0.95; hand-chosen threshold for high-confidence classification.
  • Strong address join threshold = 0.9
    Gate 5 requires strong join >= 0.9; hand-chosen.
  • Top-K shortlist size = 3
    Models asked for 3 candidates; K=3 chosen for the analyst-assistance workflow.
axioms (4)
  • domain assumption DWARF debug information accurately records source function names and address ranges
    Stage 2a uses DWARF as oracle; if DWARF is wrong, ground truth is wrong.
  • domain assumption Tree-sitter parse of preprocessed source yields a complete, correct syntactic function inventory
    Stage 2b source-side inventory; parser errors would propagate into alignment.
  • domain assumption Compiler-emitted line directives correctly bridge source and binary locations
    Stage 3 alignment uses line directives as a bridge; if directives are misleading, alignment is wrong.
  • domain assumption The stable source-anchored key identifies the same logical function across optimization levels
    Section 3.4 paired analysis relies on this; if the key mismatches, the paired conclusion is invalid.

pith-pipeline@v1.3.0-alltime-deepseek · 9738 in / 8880 out tokens · 78594 ms · 2026-08-02T08:09:12.376814+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.

Figures

Figures reproduced from arXiv: 2607.07738 by Andreas U. Schmidt, Nicolas Koller.

Figure 1
Figure 1. Figure 1: The six-stage Reforge pipeline. Two independent oracles produce ground truth that is aligned, joined to decompiler output, gated, and submitted to LLMs for scoring The six stages are as follows. Stage 1, the synthetic builder, compiles each program across a controlled build matrix and emits a provenance-anchored build receipt. Stage 2 runs two independent oracles: a DWARF oracle (2a) that extracts function… view at source ↗
Figure 1
Figure 1. Figure 1: The six-stage Reforge pipeline. Two independent oracles produce ground truth that is aligned, joined to decompiler output, gated, and submitted to LLMs for scoring. 3. THE REFORGE METHOD 3.1 Design Principles and Pipeline Overview Reforge is a modular pipeline that transforms synthetic C programs through six versioned processing stages, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: High-confidence yield by optimization level. The evaluable substrate contracts by more than twenty percentage points as optimization increases This attrition is not merely a loss of sample size; it confounds the most common comparison in the literature. Comparing aggregate mean accuracy across optimization levels compares different populations of functions, because those that survive to the highest level a… view at source ↗
Figure 3
Figure 3. Figure 3: Tier transition matrices across consecutive optimization levels (O0 to O1, O1 to O2, O2 to O3). Each cell counts functions moving from a row tier at the lower level to a column tier at the higher level; the absent row and column denote exit from or entry to the evaluable set. The heavy diagonal mass shows that most surviving functions retain their tier, while attrition is concentrated in the absent column … view at source ↗
Figure 3
Figure 3. Figure 3: Tier transition matrices across consecutive optimization levels (O0 to O1, O1 to O2, O2 to O3). Each cell counts functions moving from a row tier at the lower level to a column tier at the higher level; the absent row and column denote exit from or entry to the evaluable set. The heavy diagonal mass shows that most surviving functions retain their tier, while attrition is concentrated in the absent column … view at source ↗
Figure 4
Figure 4. Figure 4: Fraction of functions whose Token F1 improves when the best of three candidates is scored instead of the single [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Structural profile of the three confidence tiers, pooled over optimization levels (log scale; n per tier shown). [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fraction of functions whose Token F1 improves [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 1 linked inside Pith

  1. [1]

    Anthropic. (2025). Disrupting the first reported AI -orchestrated cyber espionage campaign [Threat intelligence report]. https://assets.anthropic.com/m/ec212e6566a0d47/original/Disrupting-the-first-reported-AI-orchestrated-cyber- espionage-campaign.pdf Benoit, T., Wang, Y., Dannehl, M., & Kinder, J. (2025). BLens: Contrastive captioning of binary function...

    2025

  2. [22]

    4327–4343)

    (pp. 4327–4343). Available at https://www.usenix.org/system/files/sec22-chen-qibin.pdf Dramko, L., Lacomis, J., Schwartz, E. J., Vasilescu, B., & Le Goues, C. ( 2024). A taxonomy of C decompiler fidelity issues. In 33rd USENIX Security Symposium (USENIX Security

    2024

  3. [24]

    379 –396)

    (pp. 379 –396). Available at https://www.usenix.org/system/files/usenixsecurity24-dramko.pdf Ghimire, A., Zhang, J., Lingala, S., Alsulami, F., & Amsaad, F. (2025). A survey on application of AI on reverse engineering for software analysis and security. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3593456 Jiang, L., Jin, X., & Lin, Z. (2025). Beyond c...

    arXiv 2025

  4. [25]

    J., Le Goues, C., Neubig, G., & Vasilescu, B

    Available at https://www.usenix.org/system/files/usenixsecurity25-benoit.pdf Chen, Q., Lacomis, J., Schwartz, E. J., Le Goues, C., Neubig, G., & Vasilescu, B. (2022). Augmenting decompiler output with learned variable names and types. In 31st USENIX Security Symposium (USENIX Security

    2022

  5. [2021]

    (pp. 48–57). https://doi.org/10.18653/v1/2021.nlp4prog-1.6 Shang, X., Chen, G., Cheng, S., Wu, B., Hu, L., & Li, G., Zhang, W., Yu, N. (2025). BinMetric: A comprehensive binary analysis benchmark for large language models. arXiv:2505.07360. https://doi.org/10.48550/arXiv.2505.07360 Soni, V., Dutcher, A., Bao, T., & Wang, R. (2025). Benchmarking binary typ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.07360 2021