REVIEW 2 major objections 6 minor 31 references
Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries
T0 review · 2 major / 6 minor · reviewed 2026-07-08 · glm-5.2
Pith's one-line read Fine-tuning hurts decompilation; surface metrics mislead
desk verdict Solid empirical study; metric divergence finding is the real contribution; cross-lingual finding is confounded but honestly scoped read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is metric divergence: CodeBLEU and compile@k measure static properties (token overlap, AST structure, data-flow graph similarity, compilation success) that can improve under fine-tuning even as pass@k (functional correctness verified by unit tests) regresses. The paper traces this to the objective-function disconnect—cross-entropy loss optimizes local token likelihood, not program semantics—combined with the fact that single-token errors (off-by-one, wrong operator) catastrophically fail pass@k while leaving AST and data-flow graphs nearly intact.
What would settle it
Run the same six-variant fine-tuning protocol with optimization-matched Swift data (compiled at -O2 or -Osize rather than -O0). If the 4B cross-lingual interference effect disappears or substantially shrinks, the finding is attributable to optimization mismatch rather than language interference. Separately, if a broader single-family scale sweep (1.5B through 30B) with per-scale hyperparameter tuning shows no capacity-dependent regression pattern, the fine-tuning-hurts-stronger-bases claim weakens from empirical pattern to artifact of a specific configuration.
Extended reading notes
Core claim
The paper's core discovery is a triple negative result grounded in paired statistical tests. First, task-specific fine-tuning does not improve functional correctness for neural decompilation: across six configurations, none achieves a significant pass@k gain, and the strongest base model suffers a catastrophic regression. Second, surface metrics (CodeBLEU and compile@k) can diverge sharply from pass@k—improving significantly while functional correctness regresses—because fine-tuning teaches the model to produce syntactically plausible Dart that lacks correct semantics. Third, assembly sequence length is the strongest predictor of decompilation difficulty (p=0.001), with a capability cliff at
Load-bearing premise
The cross-lingual interference finding depends on the Dart and Swift training datasets being comparable because they are token-matched to within 2%, but the Dart data uses AOT-optimized binaries while Swift uses unoptimized (-O0) compilation. This optimization mismatch is a confounding factor: the observed interference could be partially or wholly due to distributional mismatch between optimized and unoptimized assembly rather than linguistic interference between Dart and
Editorial extensions
If this is right
- Any neural decompilation or code generation study reporting only CodeBLEU or compile@k may be presenting a false signal of improvement; pass@k with unit tests should be the primary metric.
- Practitioners should not assume task-specific fine-tuning improves a capable base model without execution-based verification; the strongest base model tested regressed by 5.65 pp after fine-tuning.
- The 200-instruction assembly length cliff suggests that current small-model approaches cannot handle functions beyond a certain complexity, pointing toward hierarchical or chunked decompilation as a necessary architectural direction.
- Cross-lingual training data from a different programming language can degrade decompilation performance at small scales (4B), though this interference attenuates with model capacity.
Reading between the lines
- The metric divergence finding likely extends to code translation, code repair, and code summarization tasks—any setting where fine-tuning targets surface similarity to reference code. If so, reported improvements on these tasks measured only by BLEU or CodeBLEU could mask functional regressions.
- The capacity-dependent fine-tuning regression (stronger bases suffer more from fine-tuning) is consistent with an overwriting mechanism where fine-tuning replaces general reasoning with narrow task-specific patterns. If this holds at larger scales, it would suggest that instruction-tuned or RLHF-tuned models may be better starting points for decompilation than further supervised fine-tuning.
- The complementary task-coverage pattern (no task solved by all models; 7 tasks solved by exactly one model) suggests an ensemble of diverse base models could expand the solvable frontier by roughly 33% beyond the best single model, though at increased computational cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a systematic empirical study of fine-tuning effectiveness and metric validity for neural decompilation of Dart AOT binaries. Six fine-tuned model variants across three base architectures (4B–8B parameters) are evaluated using CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. The principal findings are: (1) no fine-tuning configuration produces a statistically significant pass@k improvement, with the strongest base (Qwen3-8B) suffering a highly significant regression (-5.65 pp, p<0.001); (2) cross-lingual interference from Swift training is significant at 4B but attenuates at 8B; and (3) surface metrics (CodeBLEU, compile@k) can improve significantly while pass@k moves in the opposite direction. The paper contributes the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU implementation, and paired statistical analyses (McNemar exact tests, bootstrap CIs, Cohen's d) across all comparisons.
Significance. The paper makes a valuable methodological contribution to the neural decompilation literature. The metric divergence finding (Finding 3) is the strongest and most generalizable claim: the v5 case study, where CodeBLEU barely moved (0.746 vs. 0.767) and compile@k was nearly unchanged while pass@1 collapsed from 6.36% to 0.71% (p<0.001, 0 tasks gained, 22 lost), is a clean and compelling demonstration that surface metrics can mask functional regressions. This finding alone has implications beyond decompilation for any LLM code generation task using superficial similarity metrics. The honest reporting of non-significance (v1: p=0.21) and significant regressions (v5: p<0.001) is commendable. The release of HumanEval-Dart, the Dart-adapted CodeBLEU, model weights, and evaluation harness as a replication package strengthens reproducibility. The paired task-level statistical methodology (McNemar exact tests with bootstrap CIs, Cohen's d effect sizes, Wilcoxon signed-rank tests) is rigorous and sets a good standard for the field.
major comments (2)
- §4.5.3 (pass@k evaluation): The decoding parameters for pass@k evaluation are not specified. Section 4.5.1 states CodeBLEU used temperature 0.2, top-p 0.99, beam size 1, but Section 4.5.3 provides no corresponding specification. Since pass@k is the paper's primary metric and the metric divergence finding (Finding 3) hinges on comparing CodeBLEU and pass@k trends, the decoding parameters for pass@k must be reported. If different parameters were used for the two metrics, this could partially explain the divergence (e.g., higher temperature for pass@k sampling would increase diversity but could reduce per-sample CodeBLEU). This is load-bearing for Finding 3 and should be clarified in revision.
- §5.4, Table 6 (Finding 2 — cross-lingual interference): The optimization mismatch between Dart (AOT) and Swift (-O0) is properly acknowledged in §8.1 and scoped to Finding 2. However, the framing of the 8B result as 'consistent with the scaling hypothesis' (§5.4, line 'becomes statistically indistinguishable from zero at 8B') is somewhat misleading given the caveat that both 8B variants are catastrophically degraded relative to the base. The paper does note this ('The apparent equivalence of v5 and v6 is between two badly-degraded models'), but the abstract and conclusion still list the scaling pattern as a principal finding. The authors should more clearly flag in the abstract and conclusion that the 8B interference equivalence is between two severely regressed models, not healthy ones, so readers do not over-interpret the scaling claim.
minor comments (6)
- §4.5.1: The CodeBLEU test set (73 functions) and pass@k test set (154 tasks) are different datasets. A brief note explaining why different test sets were used for different metrics would help readers understand the evaluation design.
- Table 3: The Qwen3-Max reference row reports pass@1=18.38% and pass@5=25.29% but leaves CodeBLEU, compile@1, and compile@5 as '—'. A brief note explaining why these metrics were not computed for the reference model would be helpful.
- §5.7, Table 8: The bin labels use '<50', '50–100', '100–200', '200+'. It would aid readability to also include the count of tasks in each bin (which is provided in the 'n' column but easy to miss).
- §8.3: The paper mentions 'two independent inference runs for Qwen3-8B and v5' with pass@1 values of 6.36%/6.10% and 0.71%/1.10%. It would strengthen the paper to report the statistical analysis using both runs or at least note whether the alternative run changes any significance conclusions.
- The 'Data Availability' section appears twice (identical text), likely a formatting artifact.
- §4.1.2: The mention of 'chain-of-thought reasoning traces from DeepSeek-R1' in the synthetic data is intriguing but underexplained. A brief note on how these traces were incorporated into training (as prefixes? as separate fields?) would improve reproducibility.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. Both major comments identify genuine gaps in the manuscript that we will address in revision. Comment 1 (missing pass@k decoding parameters) is a straightforward omission we will correct. Comment 2 (scaling claim framing in abstract/conclusion) is a fair point about over-interpretation risk that we will fix by adding the caveat to the abstract and conclusion.
read point-by-point responses
-
Referee: §4.5.3 (pass@k evaluation): The decoding parameters for pass@k evaluation are not specified. Section 4.5.1 states CodeBLEU used temperature 0.2, top-p 0.99, beam size 1, but Section 4.5.3 provides no corresponding specification. Since pass@k is the paper's primary metric and the metric divergence finding (Finding 3) hinges on comparing CodeBLEU and pass@k trends, the decoding parameters for pass@k must be reported. If different parameters were used for the two metrics, this could partially explain the divergence.
Authors: The referee is correct that this is an omission. The pass@k evaluation used the same decoding parameters as CodeBLEU: temperature 0.2, top-p 0.99, beam size 1, with n=10 independent samples per task. We will add this specification to §4.5.3 in the revised manuscript. Because the same decoding configuration was used for both CodeBLEU and pass@k, the metric divergence finding (Finding 3) cannot be attributed to a decoding parameter mismatch. We agree this is load-bearing for the paper's central claim and appreciate the referee flagging it. revision: yes
-
Referee: §5.4, Table 6 (Finding 2 — cross-lingual interference): The framing of the 8B result as 'consistent with the scaling hypothesis' is somewhat misleading given the caveat that both 8B variants are catastrophically degraded relative to the base. The paper does note this ('The apparent equivalence of v5 and v6 is between two badly-degraded models'), but the abstract and conclusion still list the scaling pattern as a principal finding. The authors should more clearly flag in the abstract and conclusion that the 8B interference equivalence is between two severely regressed models, not healthy ones, so readers do not over-interpret the scaling claim.
Authors: We agree. The body text (§5.4) and conclusion already contain the caveat that the 8B equivalence is between two badly-degraded models, but the abstract does not. This creates a risk that readers who read only the abstract will over-interpret the scaling claim. In the revised manuscript, we will add an explicit qualifier to the abstract's second finding, along the lines of: 'cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis—though this 8B equivalence is between two configurations that have both already suffered catastrophic fine-tuning regression relative to the base.' We will make a corresponding adjustment to the conclusion to ensure the caveat is prominent there as well. The body text in §5.4 already states this clearly and will remain as-is. revision: yes
Circularity Check
No significant circularity; the paper is a self-contained empirical study with one non-load-bearing self-citation to prior work.
full rationale
This is an empirical study with no formal derivation chain. The three principal findings (no significant pass@k improvement, cross-lingual interference scaling, metric divergence) are all grounded in independent measurements: fine-tuned models are evaluated against their own base models on separate test sets (HumanEval-Dart for pass@k, held-out 73-sample set for CodeBLEU, 126-function set for compile@k), using standard statistical tests (McNemar exact, Wilcoxon signed-rank, bootstrap CIs). No metric is defined in terms of another in a way that would create circularity. The paper cites its own prior work [Abualazm and Abo Elhassan 2026] in Section 1.5, but this citation is contextual (establishing that the present study extends prior work by adding pass@k, a third architecture, and statistical tests), not load-bearing for any central claim. The metric divergence finding (Finding 1) is demonstrated through within-model comparisons (v3 vs. its base, v5 vs. its base) under identical evaluation conditions, and is independently corroborated by DecompileBench [Gao et al. 2025] arriving at a similar conclusion from a different angle. The cross-lingual interference finding (Finding 2) properly acknowledges its optimization-mismatch confound rather than circularly assuming it away. No fitted parameter is renamed as a prediction, no uniqueness theorem is invoked, and no ansatz is smuggled through self-citation. The self-citation warrants a score of 1 rather than 0, but it does not undermine any claim.
Assumptions & free parameters
free parameters (7)
- LoRA rank r =
32
- LoRA alpha =
32
- LoRA dropout =
0.09
- Learning rate =
2e-5
- Label smoothing =
0.1
- Decoding temperature =
0.2
- Win/loss/tie margin =
0.01
assumptions (5)
- domain assumption HumanEval-Dart is a representative benchmark for Dart decompilation difficulty
- domain assumption Token-matched datasets isolate the effect of language choice from data volume
- standard math pass@k with unit tests is the gold standard for functional correctness
- standard math McNemar's exact test is appropriate for paired task-level binary outcome comparison
- domain assumption LoRA/DoRA fine-tuning with rank 32 is a reasonable configuration for 4B-8B models
Cite this review
Pith. "Pith review of Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries." pith.science (2026). https://pith.science/paper/J353I7LK
@misc{pith2026260706125,
author = {Pith},
title = {Pith review of: Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries},
year = {2026},
howpublished = {\url{https://pith.science/paper/J353I7LK}},
note = {Machine review of arXiv:2607.06125}
}
read the original abstract
Neural decompilation is increasingly studied as a code-generation problem, yet its evaluation methodology remains underdeveloped for modern languages. We present a systematic empirical study of fine-tuning effectiveness and metric validity for Dart Ahead-of-Time (AOT) neural decompilation. We evaluate six fine-tuned model variants across three base architectures (4B-8B parameters) using three metrics: CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. Our study yields three principal findings grounded in paired task-level statistical tests. First, no fine-tuning configuration produces a statistically significant pass@k improvement. The sole positive case yields +0.71 pp (McNemar p=0.21), while fine-tuning the strongest base (Qwen3-8B) causes a highly significant regression of -5.65 pp (p<0.001). This capacity-dependent trend is consistent across architectures but needs broader scale sweeps. Second, cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis. Third, we demonstrate metric divergence: CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction. This has implications for any LLM code generation task where fine-tuning targets superficial similarity. Error analysis reveals assembly sequence length is the strongest predictor of task difficulty (p=0.001), with a capability cliff at 200 instructions. We contribute the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU, and empirical evidence that pass@k must be the primary evaluation metric for neural decompilation.
Figures
Reference graph
Works this paper leans on
-
[1]
R. Abualazm and A. Abo Elhassan, ``LLMs as Idiomatic Decompilers: Recovering High-Level Code from x86-64 Assembly for Dart,'' in Proc. SANER ERA Track, 2026. arXiv:2604.02278
-
[2]
Armengol-Estap\'e et al., ``SLaDe: A Portable Small Language Model Decompiler,'' in Proc
J. Armengol-Estap\'e et al., ``SLaDe: A Portable Small Language Model Decompiler,'' in Proc. CGO, 2024
work page 2024
-
[3]
B. Athiwaratkun et al., ``Multi-lingual Evaluation of Code Generation Models,'' in NeurIPS Datasets and Benchmarks, 2023
work page 2023
-
[4]
Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study
R. Baltaji et al., ``Cross-lingual Transfer in Programming Languages,'' arXiv:2310.16937; TMLR, 2025
work page Pith review arXiv 2025
-
[5]
A. Boronat and J. Mustafa, ``MDRE-LLM: A Tool for Analysing and Applying LLMs in Software Reverse Engineering,'' in Proc. SANER Tool Demo Track, 2025
work page 2025
-
[6]
Evaluating Large Language Models Trained on Code
M. Chen et al., ``Evaluating Large Language Models Trained on Code,'' arXiv:2107.03374, 2021
work page Pith review arXiv 2021
-
[7]
Choi et al., ``Toward Efficient Deobfuscation via Large Language Models,'' IEEE Access, 2026
B. Choi et al., ``Toward Efficient Deobfuscation via Large Language Models,'' IEEE Access, 2026
work page 2026
-
[8]
Cifuentes, ``Reverse Compilation Techniques,'' Ph.D
C. Cifuentes, ``Reverse Compilation Techniques,'' Ph.D. dissertation, Queensland Univ. of Technology, 1994
work page 1994
Show all 31 references
-
[9]
DeepSeek Team, ``DeepSeek-R1: Reasoning-First Language Models,'' 2025
2025
-
[10]
Dramko et al., ``A Taxonomy of C Decompiler Fidelity Issues,'' in Proc
L. Dramko et al., ``A Taxonomy of C Decompiler Fidelity Issues,'' in Proc. USENIX Security, 2024
2024
-
[11]
Dramko, C
L. Dramko, C. Le Goues, and E. J. Schwartz, ``Idioms: Neural Decompilation with Joint Code and Type Definition Prediction,'' arXiv:2502.04536, 2025
2025 arXiv
-
[12]
Feng et al., ``Self-Constructed Context Decompilation with Fine-grained Alignment Enhancement,'' in Findings of EMNLP, 2024
Y. Feng et al., ``Self-Constructed Context Decompilation with Fine-grained Alignment Enhancement,'' in Findings of EMNLP, 2024
2024
-
[13]
Gao et al., ``DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios,'' in Findings of ACL, 2025
Z. Gao et al., ``DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios,'' in Findings of ACL, 2025
2025
-
[14]
Hosseini and B
I. Hosseini and B. Dolan-Gavitt, ``Beyond the C: Retargetable Decompilation using NMT,'' in Proc. BAR, 2022
2022
-
[15]
E. J. Hu et al., ``LoRA: Low-Rank Adaptation of Large Language Models,'' in Proc. ICLR, 2022
2022
-
[16]
Hu et al., ``DeGPT: Optimizing Decompiler Output with LLM,'' in Proc
P. Hu et al., ``DeGPT: Optimizing Decompiler Output with LLM,'' in Proc. NDSS, 2024
2024
-
[17]
Hu et al., ``SoK: Potentials and Challenges of Large Language Models for Reverse Engineering,'' arXiv:2509.21821, 2025
X. Hu et al., ``SoK: Potentials and Challenges of Large Language Models for Reverse Engineering,'' arXiv:2509.21821, 2025
2025
-
[18]
Jelodar, M
H. Jelodar, M. Meymani, and R. Razavi-Far, ``Large Language Models (LLMs) for Source Code Analysis: Applications, Models and Datasets,'' arXiv:2503.17502, 2025
2025 arXiv
-
[19]
Jiang, C
N. Jiang, C. Wang, K. Liu, X. Xu, L. Tan, and X. Zhang, ``Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning,'' in Proc. ICLR, 2025. arXiv:2311.13721
2025
-
[20]
Kaplan et al., ``Scaling Laws for Neural Language Models,'' arXiv:2001.08361, 2020
J. Kaplan et al., ``Scaling Laws for Neural Language Models,'' arXiv:2001.08361, 2020
2001 arXiv
-
[21]
D. S. Katz et al., ``Using Recurrent Neural Networks for Decompilation,'' in Proc. SANER, 2018
2018
-
[22]
Lacomis et al., ``DIRE: A Neural Approach to Decompiled Identifier Naming,'' in Proc
J. Lacomis et al., ``DIRE: A Neural Approach to Decompiled Identifier Naming,'' in Proc. ASE, 2019
2019
-
[23]
Liu et al., ``DoRA: Weight-Decomposed Low-Rank Adaptation,'' in Proc
S. Liu et al., ``DoRA: Weight-Decomposed Low-Rank Adaptation,'' in Proc. ICML, 2024
2024
-
[24]
Manuel et al., ``Enhancing Reverse Engineering: Investigating and Benchmarking Large Language Models for Vulnerability Analysis in Decompiled Binaries,'' arXiv:2411.04981, 2024
D. Manuel et al., ``Enhancing Reverse Engineering: Investigating and Benchmarking Large Language Models for Vulnerability Analysis in Decompiled Binaries,'' arXiv:2411.04981, 2024
2024 arXiv
-
[25]
Muennighoff et al., ``Crosslingual Generalization through Multitask Finetuning,'' in Proc
N. Muennighoff et al., ``Crosslingual Generalization through Multitask Finetuning,'' in Proc. ACL, 2023
2023
-
[26]
Qwen Team, ``Qwen3 Technical Report,'' 2025
2025
-
[27]
Ren et al., ``CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,'' in Proc
S. Ren et al., ``CodeBLEU: a Method for Automatic Evaluation of Code Synthesis,'' in Proc. AAAI, 2021
2021
-
[28]
Tan et al., ``LLM4Decompile: Decompiling Binary Code with Large Language Models,'' in Proc
H. Tan et al., ``LLM4Decompile: Decompiling Binary Code with Large Language Models,'' in Proc. EMNLP, 2024
2024
-
[29]
Tan et al., ``SK ^2 Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin,'' arXiv:2509.22114, 2025
H. Tan et al., ``SK ^2 Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin,'' arXiv:2509.22114, 2025
2025
-
[30]
W. K. Wong et al., ``Refining Decompiled C Code with Large Language Models,'' arXiv:2310.06530, 2023
2023 arXiv
-
[31]
Xu et al., ``Unleashing the Power of Generative Model in Recovering Variable Names from Stripped Binary,'' in Proc
X. Xu et al., ``Unleashing the Power of Generative Model in Recovering Variable Names from Stripped Binary,'' in Proc. NDSS, 2025
2025
Reviewed July 8, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.