REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

Andreas U. Schmidt; Nicolas Koller

The main obstacle to fair LLM reverse-engineering benchmarks is unreliable binary-to-source alignment under optimization, not model capability; unpaired scores hide survivorship bias.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 20:08 UTC pith:QTJ5ZJOY

load-bearing objection Solid methodological paper: the survivorship-bias result is real on their corpus, the funnel is usable, and the main limit is the synthetic scale they already flag. the 2 major comments →

arxiv 2607.07738 v2 pith:QTJ5ZJOY submitted 2026-07-07 cs.SE cs.AIcs.CRcs.PL

REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

Nicolas Koller , Andreas U. Schmidt This is my paper

classification cs.SE cs.AIcs.CRcs.PL

keywords Reverse engineeringLLMsBinary analysisBenchmarkingDecompilationFunction namingGround-truth alignment

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Claims about large language models recovering names and structure from decompiled binaries have outrun reliable measurement. The hard problem is not the models themselves but constructing trustworthy ground truth: after compilation and optimization, many source functions no longer map cleanly onto binary functions. Existing benchmarks treat that mapping as a solved preprocessing step and report accuracy without saying how many functions remained evaluable. This paper introduces Reforge, a provenance-tracked pipeline that builds function-level ground truth from C source through compilation, dual oracles, alignment, and decompilation, then scores every function with an eight-gate confidence funnel into three explicit tiers. On a controlled micro-benchmark, high-confidence yield falls from 87.2 percent to 65.9 percent as optimization rises, and the apparent drop in naming performance is largely an artifact of which functions survive rather than individual functions becoming harder. Anyone evaluating AI tools for reverse engineering, malware analysis, or security automation should care, because the measurement substrate itself determines what the numbers can mean.

Core claim

The principal obstacle to fair evaluation of LLMs on reverse-engineering tasks such as function naming is the reliability of binary-to-source alignment under compiler optimization. High-confidence yield falls from 87.2 percent to 65.9 percent across optimization levels, and unpaired aggregate comparisons overstate optimization-induced performance decay through survivorship bias; paired analysis of the same functions via a stable source-anchored key shows only marginal per-function change.

What carries the argument

The eight-gate confidence funnel: an auditable sequence of binary quality checks that converts alignment uncertainty into GOLD, SILVER, and BRONZE strata, used together with provenance-tracked ground-truth construction and source-anchored stable keys that enable paired cross-optimization analysis.

Load-bearing premise

Findings drawn from a small researcher-authored set of synthetic C programs, one compiler, one architecture, and one decompiler are representative enough to prescribe how real-world binary benchmarks should be built.

What would settle it

Compile a large corpus of real open-source C projects with multiple toolchains and decompilers; if high-confidence yield stays roughly flat across optimization levels, or if unpaired and paired naming-score analyses agree, the central methodological claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Benchmarks must report alignment yield and per-stage attrition alongside accuracy numbers.
Paired analysis over a source-anchored key becomes the default whenever optimization effects are claimed.
Decompiler tools should surface small ranked name shortlists rather than single suggestions, matching how analysts actually work.
Uncertainty-aware, tier-stratified reporting makes ground-truth quality visible instead of burying it in preprocessing.
The same pipeline can be scaled to reproducible real-world builds to test whether the attrition pattern generalizes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Survivorship bias of this form likely contaminates other binary-analysis LLM benchmarks that publish per-optimization accuracy without disclosing yield.
Because alignment failure modes shift from header ambiguity at low optimization to DWARF range fragmentation at high optimization, any multi-stage binary pipeline needs stage-specific confidence gates rather than a single filter.
If shortlist uplift is common, effort spent on better candidate ranking or calibration may deliver more practical reverse-engineering assistance than chasing higher exact-match rates.
Releasing the pipeline as an open artifact invites community stress-testing on languages and architectures beyond the original C/x86-64 setup.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid methodological paper: the survivorship-bias result is real on their corpus, the funnel is usable, and the main limit is the synthetic scale they already flag.

read the letter

The thing worth knowing is not another LLM naming score. It is that unpaired optimization comparisons on function-level RE benchmarks can look like performance decay when they are mostly population shift, and that this paper measures that cleanly on a controlled substrate.

What is new is the packaging: a provenance-tracked build-to-decompile pipeline, an eight-gate confidence funnel with GOLD/SILVER/BRONZE strata, and a source-anchored stable key that lets them run paired analysis. The numbers land: high-confidence yield drops 87.2% to 65.9% across opts, the evaluable set shrinks about 40%, tier transitions show attrition more than demotion of survivors, and paired Token-F1 deltas are tiny with non-significant Wilcoxon results. That is a concrete demonstration of survivorship bias, not a restatement of DIRTY/DIRECT/EUBA. They also ship code and data, treat decompiler and LLM stages as documented uncertainty, and correctly label the seven-model naming band (roughly 0.18–0.29 F1) as illustrative only. Citation pattern is honest; they engage Dramko, Tan, Leger, and the recent binary-LLM benchmarks without overclaiming capability.

Soft spots are real but proportionate. The corpus is 15 synthetic C programs, one compiler, one arch, one decompiler. That is enough to show the mechanism and the reporting prescription; it is not enough to claim the same yield curve on production software. Gate thresholds (overlap 0.95, strong join 0.9) are free parameters without a sensitivity sweep. Tiers mix alignment quality with structural simplicity. Shortlist uplift is an oracle upper bound. None of that breaks the central methodological claim, and Sections 3.2 and 5 already own the external-validity trade-off.

This is for people who build or consume function-level binary/LLM benchmarks and for anyone writing optimization-level comparisons. Math and stats are light but appropriate; data and attrition audit trail look solid for what they set out to do. I would send it to peer review. Engage if you care about how RE benchmarks are constructed; skim the LLM tables, read the funnel and the paired analysis.

Referee Report

2 major / 5 minor

Summary. The paper argues that fair benchmarking of LLMs on reverse-engineering subtasks (here, function naming from decompiled stripped binaries) is limited less by model capability than by unreliable binary-to-source alignment under compiler optimization. It presents Reforge, a provenance-tracked six-stage pipeline that builds function-level ground truth from synthetic C sources via controlled compilation, dual oracles (DWARF and Tree-sitter), line-directive alignment, Ghidra decompilation, and an eight-gate confidence funnel that stratifies functions into GOLD/SILVER/BRONZE tiers. On a 15-program micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels while the evaluable population shrinks ~40%; unpaired aggregate comparisons of naming performance overstate optimization-induced decay through survivorship bias, whereas paired analysis over a stable source-anchored key shows only marginal per-function Token-F1 change. A proof-of-concept evaluation of seven LLMs (Token F1 ~0.18–0.29) is presented as substrate demonstration rather than capability ranking, and shortlist (top-K) utility is noted for analyst workflows.

Significance. If the methodological claims hold, the paper supplies a concrete, auditable prescription for uncertainty-aware binary-analysis benchmarks: report alignment yield and per-stage attrition, stratify by confidence, and use paired analysis over source-anchored keys when claiming optimization effects. The open pipeline, synthetic corpus, and reproducible notebooks are genuine strengths that lower the barrier to replication and extension. The survivorship-bias demonstration (Figures 2–3 and the paired Wilcoxon results) is a useful corrective to a common reporting practice in the LLM-for-RE literature. The absolute LLM numbers are correctly scoped as illustrative; the lasting contribution is the evaluation substrate and the explicit treatment of multi-source uncertainty.

major comments (2)

Section 3.2 and Section 5 correctly flag the external-validity trade-off of a 15-program synthetic micro-benchmark (single compiler, architecture, and decompiler; 226 functions at lowest optimization). The load-bearing claim that unpaired comparisons overstate decay is demonstrated only on this corpus. For the methodological prescription to be adopted as default practice, the manuscript should either (a) add at least one real-world open-source package compiled under a reproducible recipe (as the authors themselves list as future work) or (b) more sharply bound the claim to “on controlled synthetic corpora of this class” rather than implying a general rule for production binary benchmarks. Without one of these, the central prescription rests on a single, deliberately simplified population.
Section 3.3 and Table 1: the eight-gate funnel and the free thresholds (overlap ratio ≥ 0.95, strong address join ≥ 0.9) are presented without a sensitivity analysis. Because GOLD membership and the reported yield curve (87.2% → 65.9%) depend on these cut-offs, a short ablation (e.g., yield and tier sizes under ±0.05 changes) is needed to show that the attrition pattern and the paired-vs-unpaired contrast are not artifacts of the particular thresholds chosen. The authors already list gate-threshold sensitivity as future work; elevating a minimal version into the present manuscript would strengthen the load-bearing yield claim.

minor comments (5)

Figure 1 caption and Stage 5 description: clarify whether address-overlap join is many-to-one or one-to-one after Gate 3, and how residual multi-matches are recorded in the audit trail.
Section 3.4: the stable key composition (test case, file, line, column, normalized name) is clear, but a short example of graceful degradation when a component is missing would help implementers.
Section 4.1: the 84 nominal cells / 77 populated cells and the 4,837 scored predictions are useful; a compact table of per-tier, per-optimization function counts would make the design fully transparent without requiring the reader to reconstruct them from the text.
Token F1 is correctly described as a lower bound on semantic recovery; a brief note on whether any secondary semantic metric (e.g., embedding similarity) was considered and rejected would preempt a common reviewer question.
References: ensure consistent formatting of arXiv preprints and the Anthropic threat report URL; a few entries have minor spacing or capitalization inconsistencies.

Circularity Check

0 steps flagged

No circularity: empirical yield and paired-vs-unpaired measurements do not reduce to their inputs by construction.

full rationale

REFORGE is a methods-and-measurement paper. Its load-bearing claims are (i) high-confidence alignment yield falls from 87.2% to 65.9% across optimization levels, (ii) the evaluable population shrinks ~40%, and (iii) unpaired aggregate means overstate optimization-induced naming decay relative to paired analysis over a source-anchored key (per-function Token-F1 deltas −0.006 to +0.033, corrected p ≥ 0.209). These quantities are computed from independent oracles (DWARF address ranges and Tree-sitter syntactic inventory joined by compiler line directives), an explicit eight-gate funnel that never deletes records, and a stable declaration-site key; none is defined in terms of the quantity it is said to measure, and none is a fitted parameter re-labeled as a prediction. LLM Token-F1 figures are explicitly scoped as illustrative substrate demonstration, not capability estimates. Self-citation is limited to ordinary related-work placement; there is no uniqueness theorem, ansatz, or prior result by the same authors that forces the central findings. The researcher-authored synthetic corpus is an external-validity trade-off the paper itself flags, not a circular reduction. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The load-bearing claims rest on standard compiler/debug tooling assumptions, author-chosen gate thresholds and a synthetic corpus, and the operational definition of the confidence funnel itself. No physical constants or large fitted models; free parameters are the discrete thresholds that define GOLD. Invented entities are methodological constructs with independent operational definitions inside the pipeline.

free parameters (3)

overlap_ratio_threshold = 0.95
Gate 4 requires DWARF range overlap ratio ≥ 0.95; chosen by authors to exclude fragmentation, not derived from first principles.
strong_address_join_threshold = 0.9
Gate 5 requires address-join strength ≥ 0.9; author-selected cutoff that directly controls GOLD membership.
top_K_shortlist_size = 3
K=3 candidates used for oracle uplift; design choice that bounds the reported shortlist benefit.

axioms (5)

domain assumption DWARF debug information plus Tree-sitter syntactic inventory, joined via compiler line directives, can produce a usable binary-to-source function mapping when gates pass.
Core of Stages 2–3; standard in binary analysis but known to fragment under optimization (cited Dramko, Tan).
domain assumption Ghidra headless decompilation of x86-64 stripped binaries yields sufficiently stable pseudocode for the naming task on this corpus.
Stage 4; paper notes conditional determinism and records decompiler output as an uncertainty source.
domain assumption A source-declaration key (test case, file, line, column, normalized name) uniquely and stably identifies the same function across optimization levels.
Section 3.4; enables the paired analysis that is central to the survivorship claim.
domain assumption Token F1 on lowercased sub-tokens is a valid lower-bound metric for name recovery quality.
Section 3.5; standard lexical overlap, acknowledged as missing pure semantic matches.
ad hoc to paper Researcher-authored synthetic C programs exercising inlining, header sharing, recursion, etc., isolate optimization effects without real-world confounds.
Section 3.2 deliberate external-validity trade-off; 15 programs, 226 functions at -O0.

invented entities (2)

Eight-gate confidence funnel (GOLD/SILVER/BRONZE tiers) no independent evidence
purpose: Convert fuzzy 'reliable ground truth' into an auditable sequence of binary checks and stratified reporting populations.
Defined in Section 3.3 and Table 1; operational construct, not a physical entity. independent_evidence is false because the exact gate set and thresholds are paper-specific, though each gate targets a known failure mode.
Stable cross-optimization source-anchored function key no independent evidence
purpose: Track the same source function across optimization levels when binary-side IDs change (~86% between lowest levels).
Section 3.4; composition of declaration site components. Useful engineering construct rather than a new scientific object.

pith-pipeline@v1.1.0-grok45 · 14001 in / 3372 out tokens · 29207 ms · 2026-07-10T20:08:51.478328+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) are increasingly applied to reverse-engineering tasks, and recent threat-intelligence reporting shows them operating inside live offensive-security workflows. Claims about their capability, however, outpace our ability to measure it. Existing benchmarks for LLM-assisted binary analysis treat the construction of function-level ground truth as a solved pre-processing step and report accuracy without disclosing how many functions were reliably evaluable. We argue that the principal obstacle to fair evaluation is not model capability but the reliability of binary-to-source alignment under compiler optimization. This paper presents Reforge, a provenance-tracked pipeline that constructs function-level ground truth from C source through compilation, DWARF and syntactic extraction, alignment, and decompilation, and that operationalizes alignment uncertainty as an eight-gate confidence funnel with three-tier stratification. On a controlled micro-benchmark, high-confidence yield falls from 87.2% to 65.9% across optimization levels, and unpaired comparisons overstate optimization-induced performance decay through survivorship bias. A proof-of-concept evaluation of seven contemporary LLMs on function naming demonstrates the validity of the concept and generally motivates an uncertainty-aware benchmarking practice.