{"id":"1194041d-e2e0-4d87-8f44-fb86d8c8717c","arxiv_id":"2607.06125","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuning small LLMs for Dart decompilation yields no functional improvement and surface metrics can diverge from correctness.","lead":"This paper finds that fine-tuning small language models for Dart binary decompilation does not improve functional correctness, and that surface metrics like CodeBLEU can improve while correctness regresses. It matters for anyone evaluating LLM code generation, because it warns that standard metrics can hide functional failures.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Optimization mismatch is the most load-bearing concern but is properly acknowledged and scoped to Finding 2 only; primary findings (fine-tuning regression, metric divergence) are robust to this confound.","rationale":"The reader's verdict of ACCEPT with HIGH confidence is appropriate. The paper makes three principal findings, two of which (fine-tuning regression, metric divergence) are robust to the identified concern. The optimization mismatch confound is real and properly acknowledged by the authors themselves in Sections 5.4, 8.1, and 10. It affects only Finding 2 (cross-lingual interference scaling), which is the least central of the three findings and is framed as hypothesis-generating rather than confirmatory. The paper's statistical methodology is sound: paired McNemar tests with bootstrap CIs on 154 tasks, appropriate effect size reporting (Cohen's d), and honest reporting of null results (v1: p=0.21). The metric divergence finding has independent support from DecompileBench (Gao et al. 2025), which arrived at a similar conclusion from a different angle. The paper contributes a new benchmark (HumanEval-Dart), a Dart-adapted CodeBLEU, and actionable empirical evidence. The main limitations (small sample size for small effects, single hyperparameter configuration, architecture specificity) are all acknowledged and do not undermine the central claims. The paper's careful framing—positioning itself as evaluation methodology rather than a deployment-ready system—is appropriate given the low absolute pass@k numbers. No adjustment to the verdict is warranted.","tokens_in":21366,"tokens_out":3471,"duration_ms":178968,"concrete_test":"Compile Swift training data at -O2 (matching Dart AOT optimization level) and re-run the v2 and v4 fine-tuning experiments. Recompute the 4B cross-lingual interference delta (currently -2.66pp, p<0.001). If the interference effect drops below statistical significance or reverses direction, the cross-lingual interference finding (Finding 2) is attributable to optimization mismatch rather than linguistic interference. If the effect persists at similar magnitude, the scaling hypothesis interpretation is strengthened. Additionally, report the decoding parameters used for pass@k evaluation to confirm they match CodeBLEU evaluation settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the optimization mismatch (Dart AOT vs. Swift -O0) as the weakest assumption. This confound directly undermines Finding 2 (cross-lingual interference scaling), because the observed 4B interference (-2.66pp, p<0.001) could be entirely attributable to distributional mismatch between optimized and unoptimized assembly rather than linguistic interference. The paper acknowledges this in Section 8.1 and again in Section 5.4. However, this concern is properly scoped: it does not affect Finding 1 (no significant pass@k improvement; Qwen3-8B regression) or Finding 3 (metric divergence), which are the paper's strongest and most generalizable claims. Finding 1 rests on within-Dart comparisons (Dart+Synth fine-tuning vs. base), where no optimization mismatch exists. Finding 3 is demonstrated through v3 and v5 case studies that compare a fine-tuned model against its own base under identical evaluation conditions, making it immune to the cross-lingual confound. The v5 case is particularly compelling: CodeBLEU barely moved (0.746 vs. 0.767) and compile@k was nearly unchanged, yet pass@1 collapsed from 6.36% to 0.71% (p<0.001, 0 tasks gained, 22 lost). This is a clean demonstration of metric divergence that no confound undermines. One secondary concern worth noting: the paper does not specify decoding parameters (temperature, top-p) for the pass@k evaluation in Section 4.5.3, while CodeBLEU evaluation used temperature 0.2, top-p 0.99 (Section 4.5.1). If pass@k used different sampling parameters, the metric divergence could be partially inflated. However, this would not explain the v5 case where surface metrics were essentially flat regardless of sampling. Overall, the paper is unusually self-aware about its limitations, and the central claims hold under scrutiny.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents a systematic empirical study of fine-tuning effectiveness and metric validity for neural decompilation of Dart AOT binaries. Six fine-tuned model variants across three base architectures (4B–8B parameters) are evaluated using CodeBLEU, compile@k, and pass@k on a new 154-task HumanEval-Dart benchmark. The principal findings are: (1) no fine-tuning configuration produces a statistically significant pass@k improvement, with the strongest base (Qwen3-8B) suffering a highly significant regression (-5.65 pp, p<0.001); (2) cross-lingual interference from Swift training is significant at 4B but attenuates at 8B; and (3) surface metrics (CodeBLEU, compile@k) can improve significantly while pass@k moves in the opposite direction. The paper contributes the HumanEval-Dart benchmark, a Dart-adapted CodeBLEU implementation, and paired statistical analyses (McNemar exact tests, bootstrap CIs, Cohen's d) across all comparisons.","tokens_in":21648,"tokens_out":1320,"duration_ms":169784,"significance":"The paper makes a valuable methodological contribution to the neural decompilation literature. The metric divergence finding (Finding 3) is the strongest and most generalizable claim: the v5 case study, where CodeBLEU barely moved (0.746 vs. 0.767) and compile@k was nearly unchanged while pass@1 collapsed from 6.36% to 0.71% (p<0.001, 0 tasks gained, 22 lost), is a clean and compelling demonstration that surface metrics can mask functional regressions. This finding alone has implications beyond decompilation for any LLM code generation task using superficial similarity metrics. The honest reporting of non-significance (v1: p=0.21) and significant regressions (v5: p<0.001) is commendable. The release of HumanEval-Dart, the Dart-adapted CodeBLEU, model weights, and evaluation harness as a replication package strengthens reproducibility. The paired task-level statistical methodology (McNemar exact tests with bootstrap CIs, Cohen's d effect sizes, Wilcoxon signed-rank tests) is rigorous and sets a good standard for the field.","major_comments":[{"comment":"§4.5.3 (pass@k evaluation): The decoding parameters for pass@k evaluation are not specified. Section 4.5.1 states CodeBLEU used temperature 0.2, top-p 0.99, beam size 1, but Section 4.5.3 provides no corresponding specification. Since pass@k is the paper's primary metric and the metric divergence finding (Finding 3) hinges on comparing CodeBLEU and pass@k trends, the decoding parameters for pass@k must be reported. If different parameters were used for the two metrics, this could partially explain the divergence (e.g., higher temperature for pass@k sampling would increase diversity but could reduce per-sample CodeBLEU). This is load-bearing for Finding 3 and should be clarified in revision.","section":null},{"comment":"§5.4, Table 6 (Finding 2 — cross-lingual interference): The optimization mismatch between Dart (AOT) and Swift (-O0) is properly acknowledged in §8.1 and scoped to Finding 2. However, the framing of the 8B result as 'consistent with the scaling hypothesis' (§5.4, line 'becomes statistically indistinguishable from zero at 8B') is somewhat misleading given the caveat that both 8B variants are catastrophically degraded relative to the base. The paper does note this ('The apparent equivalence of v5 and v6 is between two badly-degraded models'), but the abstract and conclusion still list the scaling pattern as a principal finding. The authors should more clearly flag in the abstract and conclusion that the 8B interference equivalence is between two severely regressed models, not healthy ones, so readers do not over-interpret the scaling claim.","section":null}],"minor_comments":[{"comment":"§4.5.1: The CodeBLEU test set (73 functions) and pass@k test set (154 tasks) are different datasets. A brief note explaining why different test sets were used for different metrics would help readers understand the evaluation design.","section":null},{"comment":"Table 3: The Qwen3-Max reference row reports pass@1=18.38% and pass@5=25.29% but leaves CodeBLEU, compile@1, and compile@5 as '—'. A brief note explaining why these metrics were not computed for the reference model would be helpful.","section":null},{"comment":"§5.7, Table 8: The bin labels use '<50', '50–100', '100–200', '200+'. It would aid readability to also include the count of tasks in each bin (which is provided in the 'n' column but easy to miss).","section":null},{"comment":"§8.3: The paper mentions 'two independent inference runs for Qwen3-8B and v5' with pass@1 values of 6.36%/6.10% and 0.71%/1.10%. It would strengthen the paper to report the statistical analysis using both runs or at least note whether the alternative run changes any significance conclusions.","section":null},{"comment":"The 'Data Availability' section appears twice (identical text), likely a formatting artifact.","section":null},{"comment":"§4.1.2: The mention of 'chain-of-thought reasoning traces from DeepSeek-R1' in the synthetic data is intriguing but underexplained. A brief note on how these traces were incorporated into training (as prefixes? as separate fields?) would improve reproducibility.","section":null}],"recommendation":"minor_revision","confidential_remarks":"The reader's report and stress-test note are well-calibrated. The optimization mismatch concern is real but properly acknowledged and scoped to Finding 2 only; it does not undermine Findings 1 or 3, which are the paper's strongest claims. The missing decoding parameters for pass@k (raised in the stress-test note) is the one issue I would want addressed before acceptance, as it is load-bearing for the metric divergence finding. I agree with the reader's ACCEPT recommendation but think minor revision is more appropriate to ensure the decoding parameters are reported and the 8B interference framing is tightened in the abstract/conclusion."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. Both major comments identify genuine gaps in the manuscript that we will address in revision. Comment 1 (missing pass@k decoding parameters) is a straightforward omission we will correct. Comment 2 (scaling claim framing in abstract/conclusion) is a fair point about over-interpretation risk that we will fix by adding the caveat to the abstract and conclusion.","responses":[{"response":"The referee is correct that this is an omission. The pass@k evaluation used the same decoding parameters as CodeBLEU: temperature 0.2, top-p 0.99, beam size 1, with n=10 independent samples per task. We will add this specification to §4.5.3 in the revised manuscript. Because the same decoding configuration was used for both CodeBLEU and pass@k, the metric divergence finding (Finding 3) cannot be attributed to a decoding parameter mismatch. We agree this is load-bearing for the paper's central claim and appreciate the referee flagging it.","revision_made":"yes","referee_comment":"§4.5.3 (pass@k evaluation): The decoding parameters for pass@k evaluation are not specified. Section 4.5.1 states CodeBLEU used temperature 0.2, top-p 0.99, beam size 1, but Section 4.5.3 provides no corresponding specification. Since pass@k is the paper's primary metric and the metric divergence finding (Finding 3) hinges on comparing CodeBLEU and pass@k trends, the decoding parameters for pass@k must be reported. If different parameters were used for the two metrics, this could partially explain the divergence."},{"response":"We agree. The body text (§5.4) and conclusion already contain the caveat that the 8B equivalence is between two badly-degraded models, but the abstract does not. This creates a risk that readers who read only the abstract will over-interpret the scaling claim. In the revised manuscript, we will add an explicit qualifier to the abstract's second finding, along the lines of: 'cross-lingual interference from Swift training is highly significant at 4B (-2.66 pp, p<0.001) but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis—though this 8B equivalence is between two configurations that have both already suffered catastrophic fine-tuning regression relative to the base.' We will make a corresponding adjustment to the conclusion to ensure the caveat is prominent there as well. The body text in §5.4 already states this clearly and will remain as-is.","revision_made":"yes","referee_comment":"§5.4, Table 6 (Finding 2 — cross-lingual interference): The framing of the 8B result as 'consistent with the scaling hypothesis' is somewhat misleading given the caveat that both 8B variants are catastrophically degraded relative to the base. The paper does note this ('The apparent equivalence of v5 and v6 is between two badly-degraded models'), but the abstract and conclusion still list the scaling pattern as a principal finding. The authors should more clearly flag in the abstract and conclusion that the 8B interference equivalence is between two severely regressed models, not healthy ones, so readers do not over-interpret the scaling claim."}],"tokens_in":21217,"tokens_out":1115,"duration_ms":50855,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The main thing to know: this paper's strongest result is the metric divergence finding — CodeBLEU and compile@k can improve significantly while pass@k moves the opposite direction. The v5 case is particularly clean: surface metrics barely moved, yet pass@1 collapsed from 6.36% to 0.71% (p<0.001, zero tasks gained, 22 lost). That finding alone is worth the paper, and it generalizes beyond decompilation to any code generation task where fine-tuning optimizes for surface similarity to references. The secondary finding — no fine-tuning configuration produces a statistically significant pass@k improvement, and the strongest base (Qwen3-8B) suffers a catastrophic regression — is also solid, though based on only three architectures and one LoRA configuration, so it's hypothesis-generating rather than a confirmed scaling law. The authors are appropriately careful about this framing, which I appreciate. They also ship code, data, model weights, and a new benchmark (HumanEval-Dart, 154 tasks). The statistical methodology is rigorous: paired McNemar exact tests with bootstrap CIs, Cohen's d, Wilcoxon signed-rank. They honestly report null results and non-significance. This is good empirical practice. The cross-lingual interference finding (Finding 2) is the soft spot. The Dart training data uses AOT-optimized binaries while Swift uses -O0, so the observed interference at 4B could be distributional mismatch rather than linguistic interference. The authors acknowledge this clearly in Sections 5.4 and 8.1 and scope it properly — it doesn't touch Findings 1 or 3, which rest on within-Dart comparisons. But it does mean Finding 2 is essentially uninterpretable as stated; the token-matching controls for data volume but not for the optimization-level confound. A reviewer should push for at least acknowledging that this finding cannot be cleanly attributed to language. One minor concern the stress test raised: decoding parameters for pass@k (Section 4.5.3) aren't specified, while CodeBLEU used temperature 0.2, top-p 0.99. If pass@k used different sampling, the metric divergence could be partially inflated. But the v5 case — where surface metrics were essentially flat regardless — makes this a secondary concern, not a load-bearing one. Overall: this is a well-executed empirical study with honest reporting and a genuinely useful finding about metric validity. The scope is narrow (Dart AOT, 4B–8B models, small benchmarks) but the paper doesn't overclaim. It deserves a serious referee. Recommend accept for peer review.","headline":"Solid empirical study; metric divergence finding is the real contribution; cross-lingual finding is confounded but honestly scoped","tokens_in":22243,"tokens_out":1129,"would_cite":true,"duration_ms":50238,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Fine-tuning hurts decompilation; surface metrics mislead","keywords":["neural decompilation","Dart AOT","pass@k","CodeBLEU","fine-tuning","metric divergence","cross-lingual interference","LLM code generation"],"falsifier":"Run the same six-variant fine-tuning protocol with optimization-matched Swift data (compiled at -O2 or -Osize rather than -O0). If the 4B cross-lingual interference effect disappears or substantially shrinks, the finding is attributable to optimization mismatch rather than language interference. Separately, if a broader single-family scale sweep (1.5B through 30B) with per-scale hyperparameter tuning shows no capacity-dependent regression pattern, the fine-tuning-hurts-stronger-bases claim weakens from empirical pattern to artifact of a specific configuration.","tokens_in":21597,"feed_emoji":"🔧","tokens_out":1284,"duration_ms":224920,"temperature":0.7,"pith_summary":"This paper asks whether fine-tuning small language models (4B–8B parameters) on assembly-to-Dart pairs improves neural decompilation of Dart AOT binaries, and whether standard evaluation metrics faithfully capture that improvement. The authors fine-tune six model variants across three base architectures using two training datasets—one same-language (Dart plus synthetic Dart) and one cross-lingual (Dart plus Swift), token-matched to within 2%—and evaluate each variant on three metrics: CodeBLEU (a surface similarity score), compile@k (syntactic validity), and pass@k (functional correctness via unit tests on a new 154-task HumanEval-Dart benchmark). The central finding is negative: no fine-tuning configuration produces a statistically significant pass@k improvement. The best case yields +0.71 percentage points (McNemar p=0.21), while fine-tuning the strongest base model (Qwen3-8B) causes a highly significant regression of -5.65 percentage points (p<0.001) with zero tasks gained. The paper then demonstrates metric divergence: CodeBLEU and compile@k can improve significantly while pass@k moves in the opposite direction, meaning surface metrics can signal progress where functional correctness has regressed. The authors attribute this to cross-entropy fine-tuning optimizing for token-level pattern matching rather than program semantics, and they recommend pass@k as the primary evaluation metric for neural decompilation.","feed_headline":"Fine-tuning hurts decompilation; surface metrics mislead","feed_subtitle":"No fine-tuning config improves pass@k; CodeBLEU and compile@k can rise while correctness falls, exposing a systemic evaluation gap.","key_machinery":"The central mechanism is metric divergence: CodeBLEU and compile@k measure static properties (token overlap, AST structure, data-flow graph similarity, compilation success) that can improve under fine-tuning even as pass@k (functional correctness verified by unit tests) regresses. The paper traces this to the objective-function disconnect—cross-entropy loss optimizes local token likelihood, not program semantics—combined with the fact that single-token errors (off-by-one, wrong operator) catastrophically fail pass@k while leaving AST and data-flow graphs nearly intact.","core_discovery":"The paper's core discovery is a triple negative result grounded in paired statistical tests. First, task-specific fine-tuning does not improve functional correctness for neural decompilation: across six configurations, none achieves a significant pass@k gain, and the strongest base model suffers a catastrophic regression. Second, surface metrics (CodeBLEU and compile@k) can diverge sharply from pass@k—improving significantly while functional correctness regresses—because fine-tuning teaches the model to produce syntactically plausible Dart that lacks correct semantics. Third, assembly sequence length is the strongest predictor of decompilation difficulty (p=0.001), with a capability cliff at","pith_inferences":["The metric divergence finding likely extends to code translation, code repair, and code summarization tasks—any setting where fine-tuning targets surface similarity to reference code. If so, reported improvements on these tasks measured only by BLEU or CodeBLEU could mask functional regressions.","The capacity-dependent fine-tuning regression (stronger bases suffer more from fine-tuning) is consistent with an overwriting mechanism where fine-tuning replaces general reasoning with narrow task-specific patterns. If this holds at larger scales, it would suggest that instruction-tuned or RLHF-tuned models may be better starting points for decompilation than further supervised fine-tuning.","The complementary task-coverage pattern (no task solved by all models; 7 tasks solved by exactly one model) suggests an ensemble of diverse base models could expand the solvable frontier by roughly 33% beyond the best single model, though at increased computational cost."],"forward_implications":["Any neural decompilation or code generation study reporting only CodeBLEU or compile@k may be presenting a false signal of improvement; pass@k with unit tests should be the primary metric.","Practitioners should not assume task-specific fine-tuning improves a capable base model without execution-based verification; the strongest base model tested regressed by 5.65 pp after fine-tuning.","The 200-instruction assembly length cliff suggests that current small-model approaches cannot handle functions beyond a certain complexity, pointing toward hierarchical or chunked decompilation as a necessary architectural direction.","Cross-lingual training data from a different programming language can degrade decompilation performance at small scales (4B), though this interference attenuates with model capacity."],"fun_headline_variants":["Fine-tuning fails to improve functional correctness in neural decompilation","Surface metrics rise while decompilation correctness falls during fine-tuning","CodeBLEU and compile@k diverge from pass@k in Dart decompilation","Assembly length predicts neural decompilation difficulty over model scale","No fine-tuning configuration yields significant pass@k gains for decompilation"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The cross-lingual interference finding depends on the Dart and Swift training datasets being comparable because they are token-matched to within 2%, but the Dart data uses AOT-optimized binaries while Swift uses unoptimized (-O0) compilation. This optimization mismatch is a confounding factor: the observed interference could be partially or wholly due to distributional mismatch between optimized and unoptimized assembly rather than linguistic interference between Dart and","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning fails to improve functional correctness in neural decompilation","Surface metrics rise while decompilation correctness falls during fine-tuning","CodeBLEU and compile@k diverge from pass@k in Dart decompilation","Assembly length predicts neural decompilation difficulty over model scale","No fine-tuning configuration yields significant pass@k gains for decompilation","Cross-lingual Swift training degrades 4B but not 8B decompilation models"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1626,"prompt_tokens":660,"completion_tokens":966,"prompt_tokens_details":null},"tokens_in":660,"tokens_out":966,"duration_ms":55899,"temperature":1.0,"reasoning_tokens":871,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T15:49:05.133329+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run the same six-variant fine-tuning protocol with optimization-matched Swift data (compiled at -O2 or -Osize rather than -O0). If the 4B cross-lingual interference effect disappears or substantially shrinks, the finding is attributable to optimization mismatch rather than language interference. Separately, if a broader single-family scale sweep (1.5B through 30B) with per-scale hyperparameter tuning shows no capacity-dependent regression pattern, the fine-tuning-hurts-stronger-bases claim weakens from empirical pattern to artifact of a specific configuration.","supporting_citations":[],"review_version":1}