{"id":"0064eae7-6544-47f9-8d37-5dc4903e3c16","arxiv_id":"2411.12454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Representing binary functions as data-dependence slices and fine-tuning slice embeddings on cross-optimization pairs raises Recall@1 over prior detectors in the reported benchmarks.","lead":"StrTune splits compiled functions into data-dependence slices and fine-tunes a neural model so slices that compute the same value look similar even when the assembly differs. The approach reports higher retrieval accuracy than current binary similarity detectors, which matters for finding known vulnerabilities in firmware and other software where source code is unavailable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out split is reported between fine-tuning pairs and retrieval functions, so the headline gains may reflect train/test overlap rather than cross-configuration generalization.","rationale":"The paper's central claim is an empirical one: StrTune outperforms state-of-the-art methods in cross-configuration function retrieval. For that claim to hold, the evaluation must be a fair test of generalization. I examined the technical machinery first: data-dependence slicing, flow types, and Siamese fine-tuning. There are legitimate soft spots there, including the claim that data dependence alone fixes instruction order within a slice, and the contrastive loss in Eq. (2) is written with cosine similarity where a distance is needed. The reader already flagged these issues. In my judgment, the most load-bearing condition is more basic: the benchmark numbers must be unconfounded. The manuscript never documents a train/validation/test split. Section III-C fine-tunes on slice pairs extracted from source statements and their O0/O3 binaries, while Section V-B performs retrieval on functions drawn from the same dataset. Without a clear statement that functions used for fine-tuning and GMN training are disjoint from query and pool functions, and ideally a project-level holdout, the reported Recall@1 numbers cannot be interpreted as cross-configuration generalization. This is not an accusation of misconduct; the authors may have used the standard split from [35]. The manuscript simply omits the information, and that omission is load-bearing because the central claim rests entirely on Table I. The proposed concrete test, rerunning with a strict function-disjoint split and a leave-one-project-out split, would settle whether the concern lands. If the numbers survive that test, the empirical claim is credible; if they do not, the paper needs substantial revision. Since the reader already conditioned acceptance on missing evaluation details, I keep the verdict CONDITIONAL rather than moving it.","tokens_in":19994,"tokens_out":12279,"duration_ms":121403,"concrete_test":"Require the authors to release the exact function and statement IDs used for RoBERTa pre-training, Siamese fine-tuning, GMN training, and the query/pool sets for every row of Table I. Then rerun x64-XO and x64-XC (pool=100 and 1,000) under a strict function-disjoint split, and additionally under a leave-one-project-out split (train on six projects, test on the seventh). If Recall@1 drops materially, for example by more than 10 points, in either split, the reported 25.3% and 22.2% gains are confounded by train/test overlap rather than explained by data-dependence slicing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests entirely on Table I. Section V-A defines the dataset but never states which functions or statements are used for RoBERTa pre-training, Siamese fine-tuning, GMN training, and which functions form the query and pool sets. Section III-C builds fine-tuning pairs by taking a C source file, selecting statements, and mapping them to O0/O3 slices; Section V-B then selects 1,000 query functions and a pool from the same compilation-configuration space. If a function (or even a statement) that was used to fine-tune slice embeddings also appears in a query or pool, the model has already seen the exact syntactic slice forms it is later asked to match. The claimed Recall@1 improvements of 25.3% and 22.2% over jTrans and GMN would then be confounded by train/test overlap, not evidence of cross-configuration generalization. The manuscript may follow the standard split from [35], but it does not say so, and the omission is load-bearing because the whole argument for robustness is the reported table.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"StrTune proposes a binary code similarity detection (BCSD) pipeline in which functions are lifted to IDA Microcode, decomposed into backward data-dependence slices that serve as graph nodes, and encoded by a RoBERTa model that is first pre-trained on masked slice tokens and then fine-tuned with a Siamese contrastive objective on slice pairs from the same source statements compiled under different configurations. A Graph Matching Network (GMN) then computes function-level similarity using the slice embeddings and four/five types of edge flows. The paper reports Recall@1/MRR results on the Marcelli et al. dataset, ablations over each design component, and a small real-world vulnerability-search study on TP-Link and NetGear firmware, claiming consistent improvements over SAFE, Zeek, GMN, VulHawk, Trex, and jTrans.","tokens_in":20234,"tokens_out":5844,"duration_ms":60460,"significance":"The core idea, using data-dependence slices rather than raw basic blocks or instruction sequences as the stable semantic unit for binary similarity, is well motivated and the ablation study in Table II shows that each proposed component contributes positively. The empirical gains over GMN and jTrans on the standard Marcelli benchmark are substantial, and the real-world firmware experiments, though small, are a useful practicality check. The main value of the paper would be in demonstrating that slice-level fine-tuning plus cross-graph attention yields robust cross-configuration matching. However, the evaluation as reported has load-bearing transparency gaps (no explicit train/test separation for the fine-tuning pairs, no seeds or variance, no code/data release), and a central equation and the headline numbers contain internal inconsistencies. The underlying approach is plausible, but the manuscript needs a major revision before the claims can be accepted.","major_comments":[{"comment":"The paper never states which functions or even source files are used to construct the Siamese fine-tuning pairs in Section III-C versus which functions form the query and pool sets in Section V-B. Since the fine-tuning pairs are generated by selecting statements from C source files and mapping them to O0/O3 slices, and the query/pool functions are drawn from the same dataset and compilation-configuration space, there is a real risk of train/test overlap: the model may have seen the exact syntactic slice forms it is later asked to match. This is load-bearing because the paper's headline claim of cross-configuration generalization rests entirely on Table I. The authors must specify the split (e.g., hold out entire functions or projects) and report results under a clearly disjoint evaluation protocol.","section":"Equation (2), Section III-C"},{"comment":"Eq. (2) defines d as the expression v1·v2/(|v1|×|v2|), which is the cosine similarity, but the contrastive loss L = Σ y d² + (1−y) max(m−d, 0)² requires d to be a distance. If d is similarity, then for y=1 the first term d² is minimized by making similar slices less similar, which is the opposite of the stated objective; if d is intended to be cosine distance, the formula should be 1 − similarity. This inconsistency is not merely typographical: it determines the gradient of the fine-tuning objective and must be corrected.","section":"Abstract, Introduction, Section V-B"},{"comment":"The abstract and introduction state that StrTune achieves Recall@1 25.3% higher than jTrans in the cross-optimization task on x64, but Section V-B reports a 23.5% improvement for that task, and Table I shows 0.836 vs. 0.677 (a relative improvement of about 23.5%, not 25.3%). The 25.3% figure is not reproducible from any entry in Table I, while the 22.2% improvement over GMN and the 154.9% over VulHawk do match. The inconsistency in a headline quantitative claim must be resolved.","section":"Section III-B"},{"comment":"The claimed stability of slices across compilation configurations depends on three unverified premises: that removing EFLAG assignments and dead register definitions never changes observable behavior, that the backward-slicing procedure yields a canonical instruction order within each slice, and that slices from the same basic block are truly data-independent. The paper supports these only with the single example in Figure 5 and with empirical end-to-end results. Since the whole method rests on these premises, a direct evaluation is warranted: for a sample of functions compiled with O0/O1/O2/O3/Os, report the fraction of matching value computations that produce identical slice sequences after alignment, or otherwise quantify slice-level stability. Without this, the mechanism claimed for the Recall@1 gains remains a hypothesis.","section":"Table I and general evaluation"},{"comment":"The experimental report gives no variance across repeated runs, no random seeds for the pre-training, fine-tuning, or structure training, and no code or data release that would allow independent verification. Given that the fine-tuning step randomly selects 10% of non-conditional statements and that all numbers are point estimates, the reported differences (e.g., 0.836 vs. 0.684 in x64-XO) may be within run-to-run noise; the authors should report at least the number of seeds and standard deviations, and ideally release the constructed graphs and fine-tuning pair identities to enable the community to audit the protocol.","section":null}],"minor_comments":[{"comment":"The text states that the graph includes 'five types of edges' whereas Section II and Figure 3 describe four flow types; since data parallel flow is subdivided into sequential parallel and jump parallel, the total count should be clarified consistently.","section":null},{"comment":"The metrics section defines MRR but Table I uses 'MRR10'; the authors should define MRR@10 (or explain that it is the standard MRR truncated at rank 10) so the reader can interpret the numbers.","section":null},{"comment":"The number of source statements selected for fine-tuning pairs is not reported; only 'randomly pick 10% of statements without conditional jump' is given, and no total count or seed is stated, so the size of the fine-tuning dataset is unknown.","section":null},{"comment":"The real-world vulnerability search uses only three to seven query functions per firmware; the rank numbers are therefore not directly comparable to the large-pool Recall@1 results and should be discussed as anecdotal evidence rather than a full evaluation.","section":null},{"comment":"The rendered text contains many instances of 'S TRTUNE' (with a space) in the running text, which is a formatting artifact; the index terms and references are otherwise complete.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is interesting and the ablations suggest the design choices matter, but the evaluation transparency issues are serious enough that the current version should not be accepted. The missing train/test split for the fine-tuning pairs is the most important technical concern: if the authors cannot demonstrate that the retrieval functions were held out from fine-tuning, the headline numbers cannot be interpreted as evidence of cross-configuration generalization. I would also encourage the editor to weigh the inconsistency in the headline jTrans improvement and the error in Eq. (2) as signals that the manuscript needs careful revision before further review. The paper fits the venue, but this is not yet at the level of reproducibility expected for a security venue with empirical ML claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a serious BCSD paper with an interesting graph representation, but the headline numbers are not yet believable because the paper never says whether the fine-tuning pairs overlap with the retrieval test set. If they overlap, the 25% gains over jTrans could be leakage, not generalization. The stress-test note is right on that.\n\nWhat's new: representing a function as a graph whose nodes are backward data-dependence slices (each slice computing one value), with four edge types that encode data/control relationships. That is a real departure from basic-block graphs. Fine-tuning RoBERTa slice embeddings on pairs of slices from the same source compiled at O0/O3, via a Siamese network, is also new in this combination. The ablation shows both slicing and fine-tuning matter: removing slicing drops Recall@1 by roughly 0.34, removing fine-tuning by roughly 0.19. That is evidence the components do something.\n\nWhere it's soft: the train/test split issue is the load-bearing omission. Section V-A describes the dataset but never states which functions are used for RoBERTa pre-training, fine-tuning, GMN training, and which are query/pool. The fine-tuning pairs are derived from C source files; the test queries come from the same projects. If any function or even statement used in fine-tuning appears in the query or pool, the model has seen exact slice forms it is later asked to match. The paper must state the split explicitly and ideally release code and data. Also no error bars, no repeated seeds, no artifacts. Eq. (2) defines d as cosine similarity but uses it as a distance in contrastive loss; that is a minor but real inconsistency. The baseline set omits recent methods they cite (CLAP, BinCola, CEBin, He et al. [40]); comparing against at least one of them would strengthen the claim. The slicing assumptions (EFLAG removal, fixed instruction order within a slice) are motivated by examples but not proven; that is a minor point for a machine learning paper.\n\nTo be fair, the paper does include a limitations section, which is more than many in this area. The central idea is plausible and the evaluation, if reproducible, would make this a solid contribution. The right fix is not rejection but demands: clarify the split, release artifacts, correct Eq. (2), add variance estimates. I would send this to peer review with a request for major revision.","headline":"The slicing idea is clever and the ablations are informative, but the unreported train/test split makes the headline gains unverified until the authors confirm that fine-tuning pairs do not leak into the retrieval test set.","tokens_in":20759,"tokens_out":3428,"would_cite":true,"duration_ms":33047,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StrTune claims that slicing binary code by data dependence turns compilation-dependent instruction syntax into stable value-computation units, and that fine-tuning those slices enables matching similar functions across compilers…","keywords":["binary code similarity detection","data dependence slicing","backward slicing","Siamese fine-tuning","graph matching network","Microcode","cross-optimization","vulnerability search"],"falsifier":"Compile a set of real-world functions at O0 and O3, slice both builds independently, and use symbolic execution to check whether every slice that corresponds to a source-level value, such as a loop index, parameter, or accumulator, has a matching slice in the other build with the same def-use chain; if a source-level computation appears in one build as two separate slices or in the other build merged into one, the canonicality premise fails.","tokens_in":1932,"feed_emoji":"🔍","tokens_out":3206,"duration_ms":99948,"temperature":0.7,"pith_summary":"StrTune claims that binary functions can be matched across compilers, optimization levels, and architectures by slicing each function along data dependencies and comparing the resulting value-computation slices. A slice is a chain of instructions that together compute one value, and it is a stable semantic unit even though instruction syntax and ordering change under different compilation configurations. To make the representation stable, StrTune pre-trains a language model on slices, fine-tunes it with a Siamese network on pairs of slices that perform the same computation but look different, and then uses a graph matching network over the sliced function graph to compute similarity. The paper reports that this pipeline outperforms state-of-the-art methods, with Recall@1 25.3% higher than jTrans and 22.2% higher than GMN for cross-optimization function retrieval in x64, plus strong results in cross-architecture and real-world vulnerability search.","feed_headline":"Slicing binaries by data dependence lifts similarity recall 25%","feed_subtitle":"StrTune matches functions across compilers and architectures by comparing value computations, not instruction text.","key_machinery":"The central object is the data-dependence slice: a set of instructions obtained by backward slicing from a value-using instruction, so that each slice is the computation chain of one value, with fixed internal order and no data dependence between slices from the same basic block. The graph representation adds four edge types, sequential flow, jump flow, data dependence flow (cross-basic-block def-use), and data parallel flow (emphasizing that independent slices have unfixed relative positions). Two-stage learning carries the empirical argument: RoBERTa is pre-trained by masked prediction over normalized Microcode tokens, then fine-tuned with a Siamese network and contrastive loss on paired slices that share computational content but differ syntactically; finally a Graph Matching Network with cross-graph attention aggregates slice embeddings into a similarity score. The slicing is what converts syntax-varying assembly into stable semantic units, and the fine-tuning is what aligns those units across compilation configurations.","core_discovery":"StrTune's central claim is that backward data-dependence slicing of IDA Microcode, at a maturity level where instructions are concise, decomposes every basic block into a set of data-independent slices, each capturing the computation of a single value. These slices, together with four flow types (sequential, jump, data dependence, and data parallel), form a function graph that is stable across compilation configurations. The paper further claims that slice-level fine-tuning with a Siamese network, pairing slices that originate from the same source-level computation but are expressed in different instruction dialects, teaches the embedding model to map semantically equivalent slices close together, so the cross-graph attention in a Graph Matching Network concentrates on genuinely similar regions of two functions. The evidence offered is a series of retrieval experiments (cross-optimization, cross-compiler, cross-architecture, mixed), ablations showing each component contributes, and a real-world vulnerability search on NetGear and TP-Link firmware where StrTune ranks vulnerable functions at or near the top.","pith_inferences":["Because the slicing is performed on the disassembler's Microcode, the method's canonicality inherits any instability in that IR; a natural test is whether Ghidra's P-code or another IR yields the same slice graphs, which the paper only mentions as a possibility, not a validated equivalence.","The fine-tuning pairs are built from source-level address mapping, so for closed-source binaries such an oracle is unavailable; an inference is that the method's cross-configuration alignment would degrade without matching source, or would need a syntactic-equivalence or symbolic-execution-based pair generator.","The paper's own false-positive analysis, such as OBJ_dup and BN_dec2bn both matching on OPENSSL_malloc and OPENSSL_free calls, suggests that retaining full call names biases similarity toward common library calls; replacing call names with callee summaries could be a direct, testable improvement that the paper does not evaluate."],"forward_implications":["Functions compiled from the same source with different compilers, optimization levels, or architectures can be retrieved with high recall because value-computation slices are more stable than raw instruction syntax.","Instruction reordering no longer hurts similarity: data-parallel flow explicitly encodes that independent slices can appear in any order, and cross-graph attention aligns slices by content and flow type rather than position.","Slice-level fine-tuning with paired O0/O3 and cross-compiler or cross-architecture slices transfers to unseen compilation settings, since the model learns the dialect mapping between syntactic forms of the same computation.","In real-world vulnerability search on firmware from NetGear and TP-Link, StrTune ranks the vulnerable functions first or near-first across x86, x64, ARM-32, and MIPS-32 query builds.","Ablations imply that the slicing step is the largest contributor to precision: replacing slice nodes with basic-block nodes lowers Recall@1 by roughly 69% in relative terms, and removing fine-tuning lowers it by about 0.2 in absolute terms."],"supporting_citations":[{"why":"Supplies the notion of collecting semantics, which the paper uses to argue that value computations are the stable semantic feature across compilation configurations.","marker":"[27]"},{"why":"Provides the formal program-dependence model that motivates backward data-dependence slicing as a way to decompose instructions into value-computation chains.","marker":"[28]"},{"why":"IDA Pro is the disassembler that produces the Microcode IR, at maturity MMAT_LOCOPT, on which all slicing and graph construction operates.","marker":"[29]"},{"why":"IDAPython scripts extract the Microcode with the def-use lists and operand types needed for backward slicing.","marker":"[30]"},{"why":"RoBERTa is the transformer language model that is pre-trained on masked slices and then fine-tuned with the Siamese network.","marker":"[31]"},{"why":"The Graph Matching Network supplies the cross-graph attention mechanism that computes function-level similarity from slice-graph pairs.","marker":"[23]"},{"why":"Provides the benchmark dataset of seven open-source projects across compilers, architectures, and optimization levels, plus the Recall@K and MRR evaluation protocol.","marker":"[35]"},{"why":"jTrans is the state-of-the-art x64 baseline that StrTune claims to beat by 25.3% in cross-optimization Recall@1.","marker":"[20]"}],"fun_headline_variants":["Data-dependence slicing sharpens binary similarity search","StrTune slices by data flow to match binaries across compilers","Backward slicing on data dependence boosts binary matching","Fine-tuned slice embeddings improve binary similarity detection","Value-based slicing makes binary similarity robust to compilation"],"cache_read_input_tokens":22912,"weakest_assumption_plain":"The whole approach assumes that backward data-dependence slicing on IDA Microcode produces a canonical, behavior-preserving decomposition of a function, meaning that instructions inside a slice keep a fixed valid order, dropping EFLAG assignments and dead register definitions never changes observable behavior, and slices from one basic block are truly independent; this assumption is motivated by examples rather than proven for all code.","fun_headline_variants_meta":{"raw":{"variants":["Data-dependence slicing sharpens binary similarity search","StrTune slices by data flow to match binaries across compilers","Backward slicing on data dependence boosts binary matching","Fine-tuned slice embeddings improve binary similarity detection","Value-based slicing makes binary similarity robust to compilation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1231,"prompt_tokens":995,"completion_tokens":236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":161}},"tokens_in":611,"tokens_out":236,"duration_ms":2659,"temperature":1.0,"reasoning_tokens":161,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:30:20.479609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile a set of real-world functions at O0 and O3, slice both builds independently, and use symbolic execution to check whether every slice that corresponds to a source-level value, such as a loop index, parameter, or accumulator, has a matching slice in the other build with the same def-use chain; if a source-level computation appears in one build as two separate slices or in the other build merged into one, the canonicality premise fails.","supporting_citations":[{"cited_title":"Abstract interpretation frameworks,","cited_arxiv_id":null,"evidence_quote":"Supplies the notion of collecting semantics, which the paper uses to argue that value computations are the stable semantic feature across compilation configurations."},{"cited_title":"A formal model of program dependences and its implications for software testing, debugging, and maintenance,","cited_arxiv_id":null,"evidence_quote":"Provides the formal program-dependence model that motivates backward data-dependence slicing as a way to decompose instructions into value-computation chains."},{"cited_title":"Ida pro","cited_arxiv_id":null,"evidence_quote":"IDA Pro is the disassembler that produces the Microcode IR, at maturity MMAT_LOCOPT, on which all slicing and graph construction operates."},{"cited_title":"Idapython documentation","cited_arxiv_id":null,"evidence_quote":"IDAPython scripts extract the Microcode with the def-use lists and operand types needed for backward slicing."},{"cited_title":"How machine learning is solving the binary function similarity problem,","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark dataset of seven open-source projects across compilers, architectures, and optimization levels, plus the Recall@K and MRR evaluation protocol."}],"review_version":1}