Pith. sign in

REVIEW 4 major objections 6 minor 81 references

Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM judges can grade decompiled-code outputs without source references, reaching 63.20% average correlation with human experts versus 35.04% for traditional metrics, and per-sample configuration routing raises this by 4.5%-24.7% at lower…

desk verdict A useful HOBRE LLM-judge benchmark and router, but the reference-free framing and headline comparison are both weaker than they look. read the letter →

arxiv 2608.07038 v1 pith:QLXMEYDF submitted 2026-08-07 cs.SE cs.AIcs.CR

Xiuwei Shang , Li Hu , Xiao Jiang , Jieke Shi , Junda He , Zhou Yang , Shaoyin Cheng , Guoqiang Chen
show 2 more authors
Weiming Zhang David Lo
This is my paper · ORCID
classification cs.SEcs.AIcs.CR
keywords binaryreverseengineeringLLM-as-a-Judgereference-freeevaluationfunctionnamerecoverycodesummarizationdecompilationoptimizationhumanjudgmentcorrelationBinJudgeBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Human-oriented binary reverse engineering turns stripped decompiled pseudocode into readable function names, summaries, and source-like code, but there is no cheap way to grade such outputs when source code and executable tests are unavailable. This paper tries to establish that LLM-as-a-Judge, where a large language model scores an output directly against the pseudocode instead of matching it to a reference string, can stand in for human experts in that reference-free setting. On a new expert-annotated benchmark called BinJudgeBench covering three reverse-engineering tasks, the paper reports average correlation with human judgment of 63.20% for LLM judges, nearly double the 35.04% average of eleven traditional match- and embedding-based metrics. It also reports that no single judge configuration is best for every task or sample, and that a lightweight router, BinJudge, which picks a configuration per sample, improves correlation with human experts by 4.5%-24.7% while cutting API cost to 0.06x-0.84x of static best configurations.

What carries the argument

The load-bearing object is the judge configuration space plus a lightweight routing policy. From 9 backbone LLMs, 3 prompting strategies (zero-shot, few-shot, chain-of-thought), and 3 decoding temperatures, the paper forms 81 judge configurations, each measured by its correlation with expert scores and its API cost. BinJudgeBench supplies the gold standard: 1,233 samples across function name recovery, binary code summarization, and decompilation optimization, scored on a 1-5 Likert scale by three reverse-engineering experts using a two-tier rubric (semantic correctness and analyst utility shared across tasks, plus one task-specific dimension). BinJudge is the router: a frozen UniXcoder encoder embeds the stripped pseudocode, a learned task embedding adds task awareness, and a multilayer perceptron outputs a preference distribution over configurations, trained by KL divergence against a temperature-softmaxed target distribution derived from a sample-level utility $U_{i,c}=-(MSE_{i,c}+\lambda\cdot Cost_{i,c})$; the cost term is log-scaled so the router optimizes correlation and expense together.

What would settle it

Re-annotate a random subset of BinJudgeBench with independent reverse-engineering professionals who see only stripped pseudocode and no source code, and recompute Kendall correlations; if LLM judges drop to near the 35% level of traditional metrics, or if BinJudge's routed configurations no longer beat static best configurations, the central claim fails. A second check is to audit the 149 hard samples where human and LLM scores diverge and see whether they share a feature that would flip the router's preferred configuration to a worse one.

Watch

Extended reading notes

Core claim

The central claim is that reliable HOBRE evaluation does not require source-code references at all: LLM judges can infer semantic correctness, analyst utility, and task-specific quality directly from the stripped pseudocode. The paper's measurements, on 1,233 expert-annotated samples, show the best LLM judges reach around 69% Kendall correlation with experts on function name recovery and 68% on binary code summarization, while eleven existing metrics average 35.04% across tasks. A configuration analysis over 81 judge setups shows rank flips between tasks and between samples, so the paper concludes that a one-size-fits-all judge does not exist. BinJudge, the proposed solution, learns to route each sample to a configuration by maximizing a human-utility objective that balances score accuracy with API cost; in five-fold cross-validation it beats every static configuration, improves Kendall correlation by 4.5%-24.7%, and spends 0.06x-0.84x the cost of static best configurations, though it still lands short of the oracle upper bound.

Load-bearing premise

The gold-standard labels in BinJudgeBench come from three author-experts scoring 1,233 samples with the original source code available for reference; if those labels are noisy, biased, or unrepresentative of real reverse analysts working without source, every correlation and every router improvement is measured against a flawed target.

Editorial extensions

If this is right

  • Researchers can grade function names, summaries, and optimized pseudocode without original source code or runnable test cases, the exact situation faced in malware and closed-source-firmware analysis.
  • Evaluation design should treat judge configuration as a tunable component: a single best static configuration leaves a measurable oracle gap that per-sample routing narrows.
  • Large-scale automated evaluation of HOBRE models becomes affordable: BinJudge's routed judging spends 0.06x-0.84x of the API cost of static best configurations.
  • BinJudgeBench provides reusable expert labels and a multi-dimensional rubric for future HOBRE model comparisons, independent of reference text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if LLM judges genuinely track expert judgment, the same utility signal could be used as a training reward for HOBRE models, turning the evaluator into an optimizer; the paper only demonstrates post-hoc scoring.
  • Editorial extension: because BinJudge's inputs are just pseudocode and task type, the routing idea should transfer to other judge-design spaces, but that transfer is untested.
  • Editorial extension: the paper's own fluency-trap analysis suggests reference-free judging over-credits fluent but unsupported outputs, especially for function names; a judge that must cite a semantic anchor before scoring might reduce this bias.
  • Editorial extension: since BinJudgeBench annotators could see the original source code, a source-blind re-annotation study would test whether the benchmark's gold standard matches what real analysts, who lack source, would consider good output.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces BinJudgeBench, a benchmark of 1,233 expert-annotated samples spanning three HOBRE tasks (function name recovery, binary code summarization, decompilation optimization), and presents a systematic study of LLM-as-a-Judge for reference-free evaluation. It compares 9 LLM judges across 3 prompting strategies and 3 decoding temperatures against 11 traditional match-based and embedding-based metrics, reporting an average correlation with human judgment of 63.20% for LLM judges versus 35.04% for traditional metrics. The paper further proposes BinJudge, a lightweight router that adaptively selects the best judge configuration per task and sample, claiming improvements of 4.5%–24.7% in correlation over static configurations at 0.06×–0.84× the API cost. The authors conclude that LLM-as-a-Judge is a scalable, high-fidelity, cost-effective evaluation alternative for HOBRE.

Significance. If the central claims hold, this work would be a notable contribution: it provides a reusable expert-annotated benchmark for a domain where evaluation is notoriously difficult, gives the first systematic comparison of LLM judges for HOBRE, and demonstrates a cost-aware routing mechanism. The release of the benchmark and code at the stated DOI supports reproducibility, and the empirical breadth (9 judge LLMs, 81 configurations, 3 tasks) is substantial. The analysis of configuration variability and the cost–correlation trade-offs is useful for practitioners. However, the gold-standard labels in BinJudgeBench are not reference-free in the sense the paper claims, and the headline comparison in Table 3 is affected by test-set configuration selection and lacks statistical inference. These issues bear directly on the validity of the main quantitative claims.

major comments (4)
  1. [Section 3.3.2 / Table 1] The benchmark is described as 'reference-free,' but the annotation protocol explicitly provides annotators with the original source code S_f and requires them to judge whether the output is 'closer to a higher-level representation of the source code S_f' (Table 1, dimension 2). Thus the gold labels measure source-informed utility, not the judgment of an analyst who, as the paper's own motivation states (Section 1), lacks source access. Every correlation in Tables 3 and 7 is therefore computed against a target construct that differs from the reference-free setting the paper claims to evaluate. The independent industrial expert check in Section 6.2 uses the same source-visible protocol and does not remedy this gap. The authors should either re-annotate a subset of samples without source access, or revise the benchmark's claims to 'source-informed human judgment' and temper the 'reference-free' terminology throughout.
  2. [Section 4.2 / Table 3] The comparison between LLM-as-a-Judge and traditional metrics is not apples-to-apples. Section 4.2 states that Table 3 'reports only the configuration (i.e., prompting strategy and decoding temperature) that achieves the highest average correlation for each LLM across the three tasks,' so each LLM's reported correlation is selected post-hoc on the very human labels used for evaluation. In contrast, the traditional-metric baseline is the unweighted average over 11 metrics (the 'Average' row), with no per-metric best-selection or tuning. Consequently, the 63.20% versus 35.04% gap conflates the effect of the evaluation paradigm with the effect of test-set configuration selection. A fairer comparison would either average LLM performance over all 81 configurations, report the best traditional metric per task, or use a validation split for configuration selection.
  3. [Section 4.2 / Abstract] The paper repeatedly claims that LLM-as-a-Judge 'substantially outperforms' traditional metrics and 'significantly outperforms' them, yet no confidence intervals, significance tests, or effect-size measures are reported for the key correlation differences. With n = 1,233 samples, the reported gaps between LLM judges (for example, Codestral at 53.30% versus DeepSeek-V3.2 at 70.87% in Table 3) may or may not be statistically reliable; the same applies to the LLM-versus-traditional comparison. The authors should report bootstrap confidence intervals for Kendall's tau and apply a paired test suitable for dependent correlations (e.g., Williams's test or a bootstrap paired difference test) for the central comparisons.
  4. [Section 4.4 / Table 5] The 'oracle gap' analysis defines the Oracle Configuration as selecting, for each sample, the configuration that minimizes squared error with the human score. This oracle is, by construction, the best achievable on the test set and cannot be approximated without access to the labels; its large gap relative to static configurations (e.g., 70.31% vs 92.95% τ for FNR) therefore partly reflects the oracle's use of the gold labels rather than intrinsic sample heterogeneity. The 'no one-size-fits-all' conclusion should be supported primarily by the cross-validated gains of BinJudge in Table 7, which are meaningful, rather than by the raw oracle gap, which is an unreachable upper bound. The current presentation overstates the strength of the negative result from the oracle comparison alone.
minor comments (6)
  1. [Section 4.1.2 / Table 3] The text says 'totaling 11 traditional automated evaluation metrics,' but Table 3 lists 12 metric rows (F1-score plus BLEU, METEOR, ROUGE-L, ChrF++, CrystalBLEU, CodeBLEU, RUBY, SentenceBERT, MoverScore, BERTScore, and CodeBERTScore). Please verify the count and adjust the text or table accordingly.
  2. [Table 3] The 'Average' row for traditional metrics is computed across tasks and metrics, but F1-score is reported only for FNR. The paper should state how the per-task averages handle missing values (e.g., whether the overall average is a simple mean of all reported cells).
  3. [Section 5.1] The utility function introduces hyperparameters λ = 0.1 and Softmax temperature T = 0.5 without any sensitivity analysis. Since BinJudge's cost–correlation trade-off directly depends on these choices, a brief sensitivity study or at least a discussion of their stability would strengthen the paper.
  4. [Section 3.3.1] The sampling strategy first draws a random sample of 385 responses and then supplements with 26 targeted samples to cover all 192 (model, architecture, optimization level) triplets. This supplemental step deviates from a simple random sample; the authors should note how this affects the representativeness of BinJudgeBench relative to the underlying 2,772,768-response pool.
  5. [Section 4.2] The 'fluency trap' analysis relies on a manually selected sub-pool of 100 'seemingly fluent' samples from the unsupported pool. The selection criteria are described narratively; providing a reproducible definition or the actual selection code would allow others to verify this finding.
  6. [Title and Abstract] The terms 'reference-free' and 'reference-free evaluation benchmark' are used prominently, but per the construct-validity issue raised above, the benchmark's annotations are source-informed. At minimum, the abstract and conclusion should qualify these claims or the benchmark should be re-annotated without source access to justify the terminology.

Circularity Check

3 steps flagged · score 6.0 of 10

The 63.20% LLM-human correlation is an in-sample best-of-configuration maximum scored against a source-informed gold standard, so the 'reference-free' evaluation claim is partly built from its own inputs.

  1. fitted input called prediction [Section 4.2 (RQ1), Table 3 note]
    "Due to space constraints, Table 3 reports only the configuration (i.e., prompting strategy and decoding temperature) that achieves the highest average correlation for each LLM across the three tasks."

    The 63.20% figure in the abstract and RQ1 answer is the average of per-LLM maxima selected on BinJudgeBench itself, with no held-out split or cross-validation for this comparison. The reported 'prediction' is therefore a post-hoc best-case fit: the configuration choice was optimized on the same human labels used as the gold standard, so the headline correlation measures the selected configuration's in-sample agreement, not its expected agreement on new HOBRE outputs.

  2. self definitional [Table 1 (Human Annotation Guideline); Sections 3.3.3 and 4.2]
    "The pseudocode's project origin, corresponding source code S_f, and compilation settings are provided for reference ... Reverse Analyst Utility: To what extent does the output O_f reduce the cognitive load for the reverse engineer? Is it closer to a higher-level representation of the source code S_f?"

    BinJudgeBench is presented as a 'reference-free evaluation benchmark', but the human gold standard is produced with the source code S_f in view, and one of the two shared rating dimensions explicitly scores closeness to S_f. The LLM judges see only stripped pseudocode and the candidate output. The 63.20% correlation is therefore agreement with a source-informed human target, not with a source-free reverse analyst, so the reference-free construct is partly defined by the very reference it claims to eliminate. The independent industrial check in Section 6.2 uses the same source-visible protocol and does not resolve this.

1 more flagged steps
  1. self definitional [Section 4.4 (RQ3), Oracle Configuration and Table 5]
    "Oracle Configuration: For each sample, we select the configuration that minimizes the squared error between its score and the human expert's rating ... Static Configuration is far below Oracle Configuration, with a significant 'Oracle Gap' existing across all three tasks."

    The oracle is defined as the per-sample argmin of squared error against the human labels, so any single fixed configuration is worse or equal to the oracle by construction. The 'no one-size-fits-all' sample-level conclusion (Oracle Gap) is thus an artifact of the oracle's definition rather than an empirical discovery; only the task-level rank correlations in Table 6 provide independent evidence. This does not invalidate BinJudge, whose 5-fold CV evaluation is a more direct test, but the gap used to motivate it is partly tautological.

full rationale

Score 6: the central empirical claim is partially circular. First, 63.20% is computed after per-LLM configuration selection on the same test labels, so it is an in-sample best-case number rather than a predictive estimate. Second, the 'reference-free' benchmark's gold standard is source-informed by design (Table 1), so the reported correlations validate agreement with a source-referenced human judgment, not a reference-free analyst. Third, the 'no one-size-fits-all' sample-level conclusion is secured by the definition of the oracle configuration. The BinJudge router comparison itself uses 5-fold cross-validation and is the most self-contained part; the fine-tuned judge baselines and cost measurements are legitimate. Self-citations (e.g., [19], [48]) are not load-bearing in the derivation. The industrial expert check in Section 6.2 uses the same source-visible protocol and therefore does not repair the construct issue.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No invented physical entities. BinJudgeBench and BinJudge are released software artifacts rather than postulated theoretical entities. The main assumptions are the reliability of the author-created gold labels, the validity of rank correlations on 411 samples per task, and the premise that RLHF-trained LLMs align with human preferences. Configuration hyperparameters lambda and softmax temperature are fixed by hand without sensitivity analysis.

free parameters (3)
  • lambda (cost weight in utility) = 0.1
    Chosen by hand to trade off MSE and API cost in U_i,c; no sensitivity analysis is reported.
  • Softmax temperature T in distribution alignment = 0.5
    Used to convert utility values into target distribution P_target; set without reported tuning.
  • High-variance annotation threshold (score range >= 2) = 2 points on 1-5 Likert scale
    Determines which samples receive consensus review and affects the gold labels, but it is a design choice.
assumptions (5)
  • domain assumption Human readability and usefulness of HOBRE outputs can be captured by a single 1-5 Likert score combining semantic correctness, analyst utility, and task-specific dimensions.
    The entire benchmark and all correlations rest on this scoring model; introduced in Table 1 and Section 3.3.2.
  • domain assumption Three author-experts with over three years of reverse engineering experience, plus a partial independent expert check, produce a reliable gold standard.
    Annotations are treated as ground truth in Section 3.3.3 and Section 4; author bias is only partially mitigated by 50 samples per task from an external expert.
  • standard math Rank correlation coefficients (Kendall's tau, Spearman) on 411 ordinal samples per task are valid meta-evaluation measures.
    Used in Section 4.1.3 without confidence intervals or corrections for tied Likert values.
  • domain assumption RLHF-trained LLMs align with human preferences, making LLM-as-a-Judge naturally suited for HOBRE evaluation.
    Stated in Section 1 and Section 2.3 as motivation; not directly tested by the paper.
  • ad hoc to paper Source code can be used by annotators as a semantic reference despite the claimed source-binary semantic mismatch.
    Annotators are given S_f in Table 1 while the paper argues source-derived references are unreliable for automated metrics; this tension is not resolved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering." pith.science (2026). https://pith.science/paper/QLXMEYDF

@misc{pith2026260807038,
  author       = {Pith},
  title        = {Pith review of: Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLXMEYDF}},
  note         = {Machine review of arXiv:2608.07038}
}
abstract

Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20\% with human judgment, outperforming traditional automated metrics at 35.04\%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5\%-24.7\% and reduces API cost to 0.06$\times$-0.84$\times$ of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.

Figures

Figures reproduced from arXiv: 2608.07038 by the authors.

Figure 1
Figure 1. Background of Human-Oriented Binary Reverse Engineering (HOBRE) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of human expert-annotated scores [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 29 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauff- mann, et al. 2024. Phi-4 technical report.arXiv preprint arXiv:2412.08905(2024)

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  4. [4]

    Ali Al-Kaswan, Toufique Ahmed, Maliheh Izadi, Anand Ashok Sawant, Premku- mar Devanbu, and Arie van Deursen. 2023. Extending source code pre-trained language models to summarise decompiled binaries. In2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 260–271. doi:10.1109/saner56733.2023.00033

    arXiv 2023

  5. [5]

    Anthropic. 2024. Claude 3.5 Sonnet. https://www.anthropic.com/news/claude- 3-5-sonnet Accessed: 2026-03-22

    work page 2024

  6. [6]

    Jordi Armengol-Estapé, Jackson Woodruff, Alexander Brauckmann, José Wesley de Souza Magalhaes, and Michael FP O’Boyle. 2022. ExeBench: an ML-scale dataset of executable C functions. InProceedings of the 6th ACM SIGPLAN Interna- tional Symposium on Machine Programming. 50–59. doi:10.1145/3520312.3534867

    arXiv 2022

  7. [7]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72. https://aclanthology.org/W05-0909/

    work page 2005

  8. [8]

    Guoqiang Chen, Huiqi Sun, Daguang Liu, Zhiqi Wang, Qiang Wang, Bin Yin, Lu Liu, and Lingyun Ying. 2025. ReCopilot: Reverse Engineering Copilot in Binary Analysis.arXiv preprint arXiv:2505.16366(2025). https://arxiv.org/html/2505. 16366v1

    arXiv 2025

Show all 81 references
  1. [9]

    Qibin Chen, Jeremy Lacomis, Edward J Schwartz, Claire Le Goues, Graham Neubig, and Bogdan Vasilescu. 2022. Augmenting decompiler output with learned variable names and types. In31st USENIX Security Symposium (USENIX Security 22). 4327–4343. https://www.usenix.org/conference/us...

    2022

  2. [10]

    Michael L Collard, Michael John Decker, and Jonathan I Maletic. 2013. srcml: An infrastructure for the exploration, analysis, and manipulation of source code: A tool demonstration. In2013 IEEE International conference on software maintenance. IEEE, 516–519. doi:10.1109/icsm.2013.85

    2013 doi

  3. [11]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic...

    2025 arXiv

  4. [12]

    Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. Code- score: Evaluating code generation by learning code execution.ACM Transactions on Software Engineering and Methodology34, 3 (2025), 1–22. doi:10.1145/3695991

    2025 doi

  5. [13]

    Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: precisely and efficiently measuring the similarity of code. InProceedings of the 37th IEEE/ACM Interna- tional Conference on Automated Software Engineering. 1–12. doi:10.1145/3551349. 3556903

    2022 doi

  6. [14]

    Han Gao, Shaoyin Cheng, Yinxing Xue, and Weiming Zhang. 2021. A lightweight framework for function name reassignment based on large-scale stripped binaries. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 607–619. doi:10.1145/346...

    2021

  7. [15]

    GNU Project. 2026. GNU FTP Site. http://ftp.gnu.org/gnu. Accessed: 2026-03-22

    2026

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024). https://arxiv.org/pdf/2407.21783v3

    2024 arXiv

  9. [17]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al . 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948(2025). https://arxiv.org/abs/...

    2025 arXiv

  10. [18]

    Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, and Martin Vechev

  11. [19]

    Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2026. LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead.ACM Transactions on Software Engineering and Methodology(2026). doi:10.114...

    2026 doi

  12. [20]

    Hex-RaysSA. 2026. "IDA Pro". https://www.hex-rays.com/products/ida

    2026

  13. [21]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2.5-coder technical report.arXiv preprint arXiv:2409.12186(2024)

    2024 arXiv

  14. [22]

    Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, and Robby T Tan. 2026. Codejudgebench: Benchmarking llm-as-a-judge for coding tasks. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 19416–19448. doi:10.1865...

    2026 doi

  15. [23]

    Xin Jin, Kexin Pei, Jun Yeon Won, and Zhiqiang Lin. 2022. Symlm: Predicting function names in stripped binaries via context-sensitive execution-aware code embeddings. InProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security. 1631–1645. doi:10.114...

    2022

  16. [24]

    Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. 2024. A survey of reinforcement learning from human feedback.Transactions on Machine Learning Research(2024). https://openreview.net/pdf?id=f7OkIurx4b

    2024

  17. [25]

    Maurice G Kendall. 1938. A new measure of rank correlation.Biometrika30, 1-2 (1938), 81–93. doi:10.2307/2332226

    1938 doi

  18. [26]

    Abhishek Kumar, Sonia Haiduc, Partha Pratim Das, and Partha Pratim Chakrabarti. 2024. LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization.arXiv preprint arXiv:2409.00630(2024). https://arxiv.org/ abs/2409.00630

    2024 arXiv

  19. [27]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al

  20. [28]

    Gangyang Li, Xiuwei Shang, Shaoyin Cheng, Junqi Zhang, Li Hu, Xu Zhu, Weim- ing Zhang, and Nenghai Yu. 2025. PseudoFix: Refactoring Distorted Structures in Decompiled C Pseudocode. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 841...

    2025

  21. [29]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out. 74–81. https://aclanthology.org/W04-1013/

    2004

  22. [30]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

    From generation to judgment: Opportunities and challenges of llm-as- a-judge. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2757–2791. doi:10.18653/v1/2025.emnlp-main.138

    2025 doi

  23. [31]

    Haolang Lu, Hongrui Peng, Guoshun Nan, Jiaoyang Cui, Cheng Wang, Weifei Jin, Songtao Wang, Shengli Pan, and Xiaofeng Tao. 2025. Malsight: Exploring malicious source code and benign pseudocode for iterative binary malware summarization.IEEE Transactions on Information Forensics...

    2025

  24. [32]

    Sebastian Lubos, Alexander Felfernig, Thi Ngoc Trang Tran, Damian Garber, Merfat El Mansi, Seda Polat Erdeniz, and Viet-Man Le. 2024. Leveraging llms for the quality assurance of software requirements. In2024 IEEE 32nd International Requirements Engineering Conference (RE). IE...

    2024

  25. [33]

    Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, et al . 2025. Deepseek- v3.2: Pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556(2025). https://arxiv.org/abs/2512.02556

    2025 arXiv

  26. [34]

    Mistral AI Team. 2025. Announcing Codestral 25.08 and the Complete Mistral Coding Stack for Enterprise. https://mistral.ai/news/codestral-25-08 Accessed: 2026-03-22

    2025

  27. [35]

    Mistral AI Team. 2025. Introducing Mistral 3. https://mistral.ai/news/mistral-3 Accessed: 2026-03-22

    2025

  28. [36]

    Jiang Ming, Meng Pan, and Debin Gao. 2012. iBinHunt: Binary hunting with inter-procedural control flow. InInternational Conference on Information Security and Cryptology. Springer, 92–109. doi:10.1007/978-3-642-37682-5_8

    2012 doi

  29. [37]

    OpenRouter. 2026. The Unified Interface For LLMs. https://openrouter.ai/ Accessed: 2026-03-22

    2026

  30. [38]

    Wendkûuni C Ouédraogo, Yinghua Li, Xueqi Dang, Pawel Borsukiewicz, Xin Zhou, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2025. Human-Aligned Code Readability Assessment with Large Language Models. arXiv preprint arXiv:2510.16579(2025). https://arxiv.org/a...

    2025

  31. [39]

    Thinh Nguyen Hung, Hai Nguyen Phuc, Khoa Tran Dinh, Nhan Le Tran Thanh, Nghia To Trong, Khoa Ngo Khanh, Duy Phan The, and Hau Pham Van. 2023. Binary Representation Embedding and Deep Learning For Binary Code Similarity Detection in Software Security Domain. InProceedings of th...

    2023

  32. [40]

    Karl Pearson. 1896. VII. Mathematical contributions to the theory of evolu- tion.—III. Regression, heredity, and panmixia.Philosophical Transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character187 (1896), 253–318. doi:10.1...

  33. [41]

    Maja Popović. 2017. chrF++: words helping character n-grams. InProceedings of the second conference on machine translation. 612–618. doi:10.18653/v1/w17-4770

    2017 doi

  34. [42]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. doi:10.3115/1073083.1073135

    2002

  35. [43]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundare- san, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis.arXiv preprint arXiv:2009.10297(2020). https://arxiv.org/abs/2009.10297

    2020 arXiv

  36. [44]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950 (2023). https://arxiv.org/abs/2308.12950

    2023 arXiv

  37. [45]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCN...

    2019 doi

  38. [46]

    Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Shikai Guo, Yanming Zhang, Weiming Zhang, and Nenghai Yu. 2025. Foc: Figure out the cryptographic func- tions in stripped binaries with llms.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 1–38. doi:10.1145/3731449

    2025 doi

  39. [47]

    Xiuwei Shang, Guoqiang Chen, Shaoyin Cheng, Benlong Wu, Li Hu, Gangyang Li, Weiming Zhang, and Nenghai Yu. 2025. BinMetric: A Comprehensive Binary Code Analysis Benchmark for Large Language Models. InProceedings of the Thirty-Fourth International Joint Conference on Artificial...

    2025 doi

  40. [48]

    Zihan Sha, Hao Wang, Zeyu Gao, Hui Shu, Bolun Zhang, Ziqing Wang, and Chao Zhang. 2025. llasm: Naming functions in binaries by fusing encoder-only and decoder-only llms.ACM Transactions on Software Engineering and Methodology 34, 4 (2025), 1–22. doi:10.1145/3702988

    2025 doi

  41. [49]

    Xiuwei Shang, Zhenkan Fu, Shaoyin Cheng, Guoqiang Chen, Gangyang Li, Li Hu, Weiming Zhang, and Nenghai Yu. 2026. An empirical study on the effectiveness of large language models for binary code understanding.Empirical Software Engineering31, 1 (2026), 1–38. doi:10.1007/s10664-...

    2026 doi

  42. [50]

    Charles Spearman. 1961. The proof and measurement of association between two things. (1961). doi:10.1037/11491-005

    1961 doi

  43. [51]

    Xiuwei Shang, Shaoyin Cheng, Guoqiang Chen, Yanming Zhang, Li Hu, Xiao Yu, Gangyang Li, Weiming Zhang, and Nenghai Yu. 2024. How Far Have We Gone in Binary Code Understanding Using Large Language Models. In2024 IEEE International Conference on Software Maintenance and Evolutio...

    2024

  44. [52]

    Iain Sutherland, George E Kalb, Andrew Blyth, and Gaius Mulley. 2006. An em- pirical examination of the reverse engineering process for binary files.Computers & Security25, 3 (2006), 221–228. doi:10.1016/j.cose.2005.11.002

    2006 doi

  45. [53]

    Hanzhuo Tan, Weihao Li, Xiaolong Tian, Siyi Wang, Jiaming Liu, Jing Li, and Yuqun Zhang. 2025. SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin.arXiv preprint arXiv:2509.22114(2025). https://arxiv.org/ abs/2509.22114

    2025

  46. [54]

    Zian Su, Xiangzhe Xu, Ziyang Huang, Kaiyuan Zhang, and Xiangyu Zhang. 2024. Source code foundation models are transferable binary analysis knowledge bases. Advances in Neural Information Processing Systems37 (2024), 112624–112655. doi:10.52202/079017-3578

    2024 doi

  47. [55]

    Weixi Tong and Tianyi Zhang. 2024. CodeJudge: Evaluating Code Generation with Large Language Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 20032–20051. doi:10.18653/v1/2024. emnlp-main.1118

    2024 doi

  48. [56]

    Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019. Does BLEU score work for code migration?. In2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 165–176. doi:10.1109/icpc. 2019.00034

    2019

  49. [57]

    Hanzhuo Tan, Qi Luo, Jing Li, and Yuqun Zhang. 2024. LLM4Decompile: De- compiling Binary Code with Large Language Models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 3473–3487. doi:10.18653/v1/2024.emnlp-main.203

    2024 doi

  50. [58]

    Martin Weyssow, Aton Kamanda, Xin Zhou, and Houari Sahraoui. [n. d.]. CodeUl- traFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences.ACM Transactions on Software Engineering and Methodology ([n. d.]). doi:10.1145/3736407

    doi

  51. [59]

    Wai Kin Wong, Huaijin Wang, Zongjie Li, Zhibo Liu, Shuai Wang, Qiyi Tang, Sen Nie, and Shi Wu. 2023. Refining decompiled c code with large language models. arXiv preprint arXiv:2310.06530(2023). https://arxiv.org/abs/2310.06530

    2023 arXiv

  52. [60]

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025. Can llms replace human evaluators? an empirical study of llm-as-a- judge in software engineering.Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1955–1977. doi:10.1145/3728963

    2025 doi

  53. [61]

    Zhengzi Xu, Bihuan Chen, Mahinthan Chandramohan, Yang Liu, and Fu Song

  54. [62]

    Zhihao Xu, Shikai Guo, Xiaochen Li, Zun Wang, and He Jiang. 2025. Simtam: Generation diversity test programs for fpga simulation tools testing via timing area mutation.ACM Transactions on Design Automation of Electronic Systems30, 2 (2025), 1–25. doi:10.1145/3705730

    2025 doi

  55. [63]

    Jiaqi Xiong, Guoqiang Chen, Kejiang Chen, Han Gao, Shaoyin Cheng, and Weim- ing Zhang. 2023. Hext5: Unified pre-training for stripped binary code information inference. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 774–786. doi:10...

    2023

  56. [64]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025). https://arxiv.org/abs/2505.09388

    2025 arXiv

  57. [65]

    Guang Yang, Yu Zhou, Xiang Chen, Wei Zheng, Xing Hu, Xin Zhou, David Lo, and Taolue Chen. 2025. Code-DiTing: Automatic Evaluation of Code Generation without References or Test Cases. In2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1...

    2025

  58. [66]

    Tong Ye, Lingfei Wu, Tengfei Ma, Xuhong Zhang, Yangkai Du, Peiyu Liu, Shouling Ji, and Wenhai Wang. 2023. CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Code. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 1...

    2023

  59. [67]

    Zhihao Xu, Shikai Guo, Guilin Zhao, Peiyu Zou, Xiaochen Li, and He Jiang

  60. [68]

    doi:10.1109/tcad.2025.3565488

    A novel HDL code generator for effectively testing FPGA logic synthesis compilers.IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems(2025). doi:10.1109/tcad.2025.3565488

    2025

  61. [69]

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embed- dings and earth mover distance. InProceedings of the 2019 conference on empirical methods in natural language processing an...

    2019 doi

  62. [70]

    Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. Code- BERTScore: Evaluating Code Generation with Pretrained Models of Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 13921–13937. doi:10.18653/v1/2023.emnlp-main.859

    2023 doi

  63. [71]

    Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luís F Gomes, Guang Yang, Kui Liu, Xin Xia, and David Lo. 2025. SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE. In2025 40th IEEE/ACM International Conference on Automated Software Engi...

    2025

  64. [72]

    Qunhong Zeng, Yuxia Zhang, Zexiong Ma, Bo Jiang, Ningyuan Sun, Klaas-Jan Stol, Xingyu Mou, and Hui Liu. 2025. Evaluating Generated Commit Messages with Large Language Models.arXiv preprint arXiv:2507.10906(2025). https: //arxiv.org/abs/2507.10906

    2025 arXiv

  65. [73]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675(2019). https://arxiv.org/abs/1904.09675

    2019 arXiv

  66. [74]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.arXiv preprint arXiv:2406.11931(2024). https://arxiv.org/abs/2406.11931

    2024 arXiv

  67. [75]

    Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. InFindings of the Association for Computational Linguistics: EACL 2024. 2232–2242. doi:10.18653/v1/2024.findings-eacl.148 Received 2026-03-26; accepted 2026-06-18

    2024 doi

  68. [78]

    Zhiping Zhou, Xiaohong Li, Ruitao Feng, Yao Zhang, Yuekang Li, Wenbu Feng, Yunqian Wang, and Yuqing Li. 2025. FidelityGPT: Correcting Decompilation Dis- tortions with Retrieval Augmented Generation.arXiv preprint arXiv:2510.19615 (2025). https://arxiv.org/abs/2510.19615

    2025

  69. [79]

    Kangchen Zhu, Zhiliang Tian, Shangwen Wang, Weiguo Chen, Zixuan Dong, Mingyue Leng, and Xiaoguang Mao. 2025. Misum: multi-modality heterogeneous code graph learning for multi-intent binary code summarization.Proceedings of the ACM on Software Engineering2, FSE (2025), 1339–136...

    2025 doi

  70. [2017]

    In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)

    Spain: security patch analysis for binaries towards understanding the pain and pills. In2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 462–472. doi:10.1109/icse.2017.49

    2017 doi

  71. [2018]

    InProceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

    Debin: Predicting debug information in stripped binaries. InProceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security. 1667–1680. doi:10.1145/3243734.3243866

    2018

  72. [2025]

    In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR)

    Can llms replace manual annotation of software engineering artifacts?. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 526–538. doi:10.1109/msr66628.2025.00086

    2025

  73. [2618]

    doi:10.1109/ase63991.2025.00214

    2025

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.