{"id":"b108dd9b-bca9-4640-8880-b7465706e7b9","arxiv_id":"2608.11766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adding instruction-level alignment, derived from compiler debug information, as an auxiliary loss improves binary code embedding quality and retrieval accuracy while giving inspectable per-instruction similarity evidence.","lead":"Binary code embedding models usually learn from function-level matches; this paper adds a finer-grained training signal that aligns individual assembly instructions across binaries compiled from the same source. The authors report improved retrieval accuracy and more interpretable similarity evidence on standard benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main result is reported without variance or significance testing; the CLAP gains in Table 2 are small (about +1 to +2 points) and could reflect run-to-run noise, so the central claim is not yet established.","rationale":"The paper is well-controlled in design: the lambda=0 baselines isolate the effect of additional training on the same debug-info subset, and RQ4 makes a good-faith attempt to bound label noise. My main concern is evidential rather than logical: the headline Table 2 result is a single realization with no variance information, and the CLAP improvement is small. The consistency across 16 cells is suggestive, but the cells share queries, candidate pools, and model weights, so the sign pattern is not an independent sign test. I therefore keep the reader's CONDITIONAL verdict: the method is plausible, but the main effect needs seed-level replication and significance testing before acceptance. The reader's label-quality concern is real but secondary; RQ4's random-noise audit and small manual audit only partially discharge it, but a failure of statistical significance would be prior to any label-quality issue.","tokens_in":24216,"tokens_out":15303,"duration_ms":170283,"concrete_test":"Run each of the four training conditions (jTrans with lambda=0, InsnAlign-jTrans, CLAP with lambda=0, InsnAlign-CLAP) with at least 5 independent seeds, evaluate the full Table 2 protocol for each seed, and report per-cell mean and standard deviation plus a paired test over queries for one representative CLAP setting. If the 95% confidence interval for InsnAlign-CLAP minus CLAP with lambda=0 includes zero, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that instruction alignment training improves function-level retrieval rests on Table 2 (RQ2), a single set of point estimates with no error bars, no repeated seeds, and no significance test. For CLAP the average gain is only 0.6590 versus 0.6396, about 1.9 points; several individual cells are under 1.5 points. The 16 settings are not independent, because query sets and candidate pools are shared across rows and columns and the underlying model is the same, so the all-16-positive pattern cannot substitute for a paired test. RQ3 reports averages over seeds 3, 5, 7, and 42 only for the discriminability analysis, not for Table 2, and RQ5's paired t-test covers synergy re-ranking, not the main InsnAlign versus lambda=0 comparison. If the Table 2 differences are within training noise, the paper's empirical claim is unsupported regardless of label quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InsnAlign, a training method that augments function-level contrastive learning for binary code embeddings with an auxiliary instruction-level alignment loss. The alignment supervision is derived from compiler debug information: assembly instructions from different compilations of the same source function are treated as aligned when they map to the same source line. The authors first show that function-level fine-tuning already improves instruction alignment in a preliminary study, then train jTrans and CLAP with the auxiliary loss on a debug-enabled subset of BinaryCorp and evaluate on BinKit. The evaluation covers instruction alignment quality, function-level Recall@1 across 16 cross-compiler/cross-optimization settings, discriminability of a mean alignment score, label-noise resilience, synergy re-ranking, and qualitative failure-mode and patch-presence case studies.","tokens_in":24387,"tokens_out":7186,"duration_ms":74465,"significance":"If the central transfer result holds, this is a useful contribution: it identifies a previously underused fine-grained supervision signal for binary code representation learning and provides an interpretable, instruction-level evidence mechanism for similarity judgments. The controlled comparison is well designed: for each base model, the λ=0 continuation and the λ>0 InsnAlign variant share dataset, tokenization, and training settings, and evaluation is performed on BinKit projects disjoint from the BinaryCorp training set. The paper also includes strengths that are easy to underweight: released code and data, a manual audit of label correctness, a quantitative analysis of failure modes, and a concrete patch-presence case study. However, the central RQ2 claim is not yet statistically established because Table 2 reports point estimates without variance, confidence intervals, or significance tests, and the only significance test in the paper concerns the synergy re-ranking mechanism, not the main λ=0 versus λ>0 comparison.","major_comments":[{"comment":"The central claim of RQ2 — that instruction alignment training improves function-level Recall@1 — is currently supported only by single point estimates. Table 2 reports no variance, confidence intervals, or significance tests for any of the 16 settings. For CLAP, the average gain is 0.0194 (0.6396 vs. 0.6590) and individual gains are between roughly +0.015 and +0.023; these differences are small enough that run-to-run noise is a plausible alternative explanation. Since RQ3 (Table 3) reports averages over seeds 3, 5, 7, and 42, the same repeated-seed protocol should be applied to the RQ2 comparison, or the authors must justify its absence. I request repeated-seed results and a paired significance test (e.g., bootstrap over query functions) for the λ=0 vs. λ>0 comparison.","section":"§5.2, Table 2"},{"comment":"The 16 settings in Table 2 are not independent replicates: they share query sets, candidate pools, and the same underlying model weights, so the statement that improvement holds 'consistently across all 16 settings' cannot substitute for a statistical test. A sign test or unanimous-vote argument over these cells would ignore the dependence structure. The only significance test in the paper, the paired t-test in §5.5 (t=27.0, p=5.2×10^{-161}), applies to the synergy re-ranking gain, not to the central InsnAlign-vs-λ=0 comparison. Consequently the main transfer claim currently lacks inferential support even though the evaluation design is otherwise carefully controlled.","section":"§5.2.2, §5.5"},{"comment":"The noise-resilience experiment corrupts instruction-to-source-line labels randomly, but the failure mode most relevant to the proposed training signal is systematic misattribution under aggressive optimization (e.g., folded lines, inlined callees absorbing line attributions). The manual audit provides useful evidence that only 2.8% of mappings are suspicious or wrong and 5.3% are plausible, but it is based on 100 sampled functions and the sample-selection procedure is not described. The paper should either specify the audit sampling and its coverage of the O3 configurations where misattribution is most likely, or add a structured-corruption experiment; otherwise the transfer result rests on label correctness that has been only partially stress-tested.","section":"§5.4"}],"minor_comments":[{"comment":"The statement 'All binaries are stripped before extracting assembly instructions' is ambiguous in light of the heavy reliance on debug information for label construction; please clarify whether stripping removes only the symbol table or also DWARF sections, and whether the debug info used for labels comes from unstripped copies of the same binaries.","section":"§4.6"},{"comment":"The glyphs '˝' and '∑' appear corrupted in the equation and the surrounding definitions; the notation should be typeset with standard sum symbols and explicit index sets.","section":"§4.2, Eq. (8)"},{"comment":"The footnote marker on 'Average improvement1' is not matched to an actual footnote; ensure the marker aligns with the explanation below the table.","section":"Table 4 footnote"},{"comment":"The sentence 'jTrans performs much better on GCC-11 compilers, indicating the training data used in this study was compiled with GCC' is an unsupported inference from the table; either provide a training-data compiler breakdown or rephrase it as a conjecture.","section":"§5.2.2"},{"comment":"The 'invisible callee semantics' example would benefit from an explicit note that symbols such as 'xcharalloc' and 'xmalloc' come from debug information rather than from the stripped binaries, to avoid confusion about how the ground truth was obtained.","section":"§5.6, Cause 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading and worth sending to reviewers, but the central empirical claim needs another round of evidence before I'd trust it fully. The new idea: use DWARF source-line mappings to train instruction-level correspondences as an auxiliary loss on top of function-level contrastive learning. That is genuinely new, at least relative to the papers I know—PalmTree has instruction embeddings but no cross-function alignment supervision, DeepBinDiff works at basic-block level without debug-info labels. The measurement study (instruction alignment as retrieval) is a sensible way to quantify the effect, and the controlled comparison is well designed: same checkpoint, same data, same settings, lambda=0 vs lambda>0. Those are the right ingredients.\n\nWhat's good: the improvement is consistent across all 16 compiler/optimization settings for both jTrans and CLAP; the effect sizes are modest (+1 to +3 Recall@1 points for CLAP, larger for jTrans), but consistent. The RQ4 label-quality analysis is more careful than most: 86.6% coverage, 86.8% correct in the manual audit, and they separate random corruption from systematic mis-attribution. The case studies are actually informative, especially the failure analysis (wrappers, truncation, near-twins) and the patch-presence heatmap. The authors ship code and data, which is creditable.\n\nSoft spot, and it's the one that matters: Table 2, the central RQ2 result, is a single set of point estimates. No error bars, no repeated seeds, no paired test. The stress-test note is right that the 16 settings are not independent—shared query pools and the same underlying model—so the all-16-positive pattern doesn't substitute for variance information. The gains for CLAP average about 1.9 points, which is within run-to-run noise for this kind of training. RQ3 reports averages over seeds only for the discriminability analysis, not for Table 2, and the t-test in RQ5 covers synergy re-ranking, not the InsnAlign vs lambda=0 comparison. So the central claim is plausible but not yet established. This is fixable: rerun with 3-5 seeds, report mean and std, do a paired test across settings or per-query. Also worth checking sensitivity to lambda; 0.001 looks like it was chosen to keep the alignment loss from dominating, but a small sweep would help.\n\nWho this is for: anyone working on binary code similarity, embedding interpretability, or using debug info as supervision. It deserves a serious referee; my verdict at this stage is conditional—accept the idea, ask for the variance/significance evidence before trusting the headline number.","headline":"Instruction alignment via debug info is a genuinely new and well-designed idea, but the main retrieval gain needs error bars before I'd believe it.","tokens_in":24911,"tokens_out":2228,"would_cite":true,"duration_ms":22904,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding instruction-level alignment supervision, derived from compiler debug information, to function-level contrastive training improves binary code similarity retrieval and makes the model's similarity judgments…","keywords":["binary code representation learning","binary code similarity analysis","instruction alignment","contrastive learning","debug information","InfoNCE","interpretable embeddings","retrieval"],"falsifier":"Retrain InsnAlign with debug labels systematically shifted one source line up or down, so every instruction is paired with a semantically unrelated counterpart while the training procedure is otherwise unchanged; if function-level Recall@1 still rises, the improvement is not caused by true instruction alignment.","tokens_in":23991,"feed_emoji":"🧩","tokens_out":8793,"duration_ms":83201,"temperature":0.7,"pith_summary":"The paper sets out to show that fine-grained instruction-level correspondences between binary functions carry a supervision signal that function-level binary code similarity models currently ignore. Using compiler debug information, two assembly instructions that trace to the same source line are treated as aligned, and the paper introduces InsnAlign, which adds an InfoNCE-style alignment loss on these correspondences to standard function-level contrastive training. The main empirical claim is that this auxiliary objective improves function-level retrieval accuracy across all 16 cross-compiler, cross-optimization settings tested, and that the resulting per-instruction alignments are more discriminative than the function embedding alone when hard negatives are compared. The paper also argues that instruction alignment makes similarity judgments interpretable, since the aligned instruction pairs serve as visible evidence for a decision. The broader point a sympathetic reader takes away is that debug information is an underused source of fine-grained labels for binary representation learning.","feed_headline":"Aligned instructions improve binary code similarity in all 16 settings","feed_subtitle":"Compiler debug info supplies fine-grained labels that lift retrieval accuracy and expose why two functions match.","key_machinery":"The central mechanism is the instruction alignment matrix $\\mathbf{M}$, built from compiler debug information: $\\mathbf{M}_{ij}=1$ when instruction $i$ in one binary and instruction $j$ in the paired binary originate from the same source line. Instruction-level embeddings are obtained by mean-pooling the transformer's token hidden states over the tokens that belong to each instruction, and a cosine-similarity matrix $\\mathbf{S}$ is computed between all instruction pairs of the two functions. The alignment objective is a symmetric, multi-positive InfoNCE loss that, for each query instruction with at least one aligned counterpart, maximizes the aggregate softmax probability over its positive matches among the paired function's instructions. This loss is added to the function-level triplet loss with a small weight $\\lambda$, chosen so that the already-converged function-level objective stays stable while the alignment signal is learned.","core_discovery":"The paper's central claim is that explicitly supervising instruction-level alignment improves the quality of function-level binary code embeddings. On the BinKit evaluation set, starting from already-finetuned jTrans and CLAP checkpoints and continuing training with the auxiliary alignment loss, InsnAlign reports higher function-level Recall@1 than the baselines in every one of the 16 compiler-optimization settings, with average gains from 0.4011 to 0.4282 for the jTrans-based model and from 0.6396 to 0.6590 for the CLAP-based model. The instruction-level signal is also more discriminative on hard negative pairs: the mean alignment score achieves a larger Cohen's d than function cosine similarity under top-5 negative sampling for both backbones. Combining the two scores in a synergy re-ranking step further improves retrieval, by 27.9% for the jTrans-based model. The paper presents this as evidence that instruction alignment is both a practical training signal and an interpretability mechanism for binary similarity decisions.","pith_inferences":["If compiler debug line mappings are widely available, instruction alignment could become a drop-in auxiliary objective for any token-level binary embedding model, not only the two transformers evaluated here.","The patch-presence-detection case suggests a concrete research direction: align a query binary against both vulnerable and patched variants and use the absence of matches on the patch's source lines as the decision signal; the paper only sketches this possibility.","A systematic label-noise audit, shifting source lines rather than corrupting them randomly, would be the natural stress test of whether the alignment objective is learning semantics or just matching debug metadata.","The failure taxonomy identifies thin wrappers, truncated contexts, and near-twin functions as the residual error classes, implying that instruction alignment will help most on functions with rich bodies and visible callee semantics."],"forward_implications":["Function-level binary similarity models can be improved with labels already present in the compilation pipeline, without collecting new function-level ground truth.","The instruction-level alignments give analysts inspectable evidence for why two functions are judged similar, which could be surfaced in reverse engineering and malware-analysis tools.","Because the gains hold across old and new compilers and across optimization levels, the benefit is not tied to a single toolchain's debug format.","The synergy between the mean alignment score and function embedding similarity suggests that fine-grained and coarse-grained signals are complementary and can be combined by re-ranking.","Alignment supervision remains effective when function-level training uses hard negative mining, so it adds value on top of stronger contrastive baselines."],"supporting_citations":[{"why":"Provides the jTrans Transformer baseline, its finetuned checkpoint, and the BinaryCorp training data reused for fair comparison.","marker":"[60]"},{"why":"Provides the CLAP baseline and its cross-modality checkpoint that InsnAlign extends.","marker":"[53]"},{"why":"Provides the BinKit evaluation dataset used to measure cross-compiler, cross-optimization Recall@1 without project overlap with training.","marker":"[26]"},{"why":"Supplies the InfoNCE loss that the instruction alignment objective is based on.","marker":"[43]"},{"why":"Supplies the supervised contrastive multi-positive generalization used when a source line expands to multiple instructions.","marker":"[25]"},{"why":"Supplies the triplet loss used as the function-level objective in the combined loss.","marker":"[48]"},{"why":"Supplies the deduplication preprocessing that removes duplicated BinKit functions before evaluation.","marker":"[55]"}],"fun_headline_variants":["Instruction alignment lifts binary retrieval in every setting","New training signal: align instructions for better binary embeddings","InsnAlign: instruction-level supervision improves binary retrieval","All 16 settings show binary retrieval gains from instruction alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on compiler debug information mapping each assembly instruction to the correct source line; if aggressive optimization systematically misattributes instructions to the wrong lines, the alignment loss trains on wrong pairings and the reported function-level gains would not follow.","fun_headline_variants_meta":{"raw":{"variants":["Instruction alignment lifts binary retrieval in every setting","New training signal: align instructions for better binary embeddings","InsnAlign: instruction-level supervision improves binary retrieval","All 16 settings show binary retrieval gains from instruction alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000467,"raw_usage":{"total_tokens":2295,"prompt_tokens":881,"completion_tokens":1414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1352}},"tokens_in":497,"tokens_out":1414,"duration_ms":10391,"temperature":1.0,"reasoning_tokens":1352,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:27:54.840811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain InsnAlign with debug labels systematically shifted one source line up or down, so every instruction is paired with a semantically unrelated counterpart while the training procedure is otherwise unchanged; if function-level Recall@1 still rises, the improvement is not caused by true instruction alignment.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the jTrans Transformer baseline, its finetuned checkpoint, and the BinaryCorp training data reused for fair comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the supervised contrastive multi-positive generalization used when a source line expands to multiple instructions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the deduplication preprocessing that removes duplicated BinKit functions before evaluation."}],"review_version":1}