{"id":"c6cb75e7-7235-4780-9d52-1acc7646510e","arxiv_id":"2608.07038","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM judges match human expert opinions on binary reverse engineering outputs better than text-similarity metrics, and a learned router improves both accuracy and cost.","lead":"This paper tests whether large language models can grade reverse-engineered code the way human experts would, and builds a benchmark and a router that picks the best grading model per sample. The work offers a cheaper, faster alternative to human review for evaluating binary code analysis tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'reference-free' benchmark is built from source-informed human labels; until LLM-human correlation is re-measured against source-free expert judgments, the 63.20% headline measures a different construct.","rationale":"The reader flagged the gold standard as the weakest assumption; I sharpen it to a specific construct-validity problem: the annotation protocol injects source code as a reference, so the benchmark validates correlation with source-informed human judgment, not the reference-free judgment the paper claims to target. I considered the selection of each LLM's best configuration on the same test labels and the absence of confidence intervals, but those are secondary: even averaging across all configurations in Table 4 keeps the LLM advantage large, and the gap is far bigger than plausible sampling error. The source-informed label issue is more foundational because it determines what all the numbers mean. The paper has real independent support, including released data, five-fold cross-validation for BinJudge, and a separate industrial expert check, so I would not reject or mark unverified; the conditional verdict remains appropriate pending a source-free annotation check.","tokens_in":27051,"tokens_out":5490,"duration_ms":51303,"concrete_test":"Re-annotate a stratified random 20% subset of BinJudgeBench (about 82 samples per task) with the same three experts, redacting the source code and project origin so annotators see only stripped pseudocode plus candidate output, using the same scoring rubric and mode/consensus protocol. Then compute Krippendorff's alpha and recompute the Table 3 correlations (LLM average vs traditional average) against these source-free labels. If the LLM-traditional gap (currently 28.16 points) collapses or alpha falls below about 0.6, the source-informed gold standard is not a valid proxy for source-free human judgment, and the headline claims must be re-scoped. If the gap and alpha hold, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"BinJudgeBench's gold standard is not reference-free. In Table 1, annotators receive the stripped pseudocode P_f and candidate O_f, but also 'the corresponding source code S_f and compilation settings are provided for reference.' The second rating dimension explicitly asks whether O_f is 'closer to a higher-level representation of the source code S_f.' So the human labels measure source-informed utility, not the judgment of an analyst confronting a stripped binary with no source. The paper's central claim is that LLM-as-a-Judge works for reference-free HOBRE evaluation, yet every reported correlation (63.20% vs 35.04% in Table 3, BinJudge's 4.5%-24.7% gains in Table 7) is computed against labels that were produced with access to the very reference the LLM judge is denied. This is a construct-validity gap, not just noise: the target variable differs from the construct the paper claims to validate. It is not resolved by the independent industrial expert check in Section 6.2, which used the same source-visible protocol. The direction of bias is unclear, so the headline gap cannot be interpreted as evidence for reference-free evaluation until the labels themselves are source-free.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BinJudgeBench, a benchmark of 1,233 expert-annotated samples spanning three HOBRE tasks (function name recovery, binary code summarization, decompilation optimization), and presents a systematic study of LLM-as-a-Judge for reference-free evaluation. It compares 9 LLM judges across 3 prompting strategies and 3 decoding temperatures against 11 traditional match-based and embedding-based metrics, reporting an average correlation with human judgment of 63.20% for LLM judges versus 35.04% for traditional metrics. The paper further proposes BinJudge, a lightweight router that adaptively selects the best judge configuration per task and sample, claiming improvements of 4.5%–24.7% in correlation over static configurations at 0.06×–0.84× the API cost. The authors conclude that LLM-as-a-Judge is a scalable, high-fidelity, cost-effective evaluation alternative for HOBRE.","tokens_in":27317,"tokens_out":7391,"duration_ms":64215,"significance":"If the central claims hold, this work would be a notable contribution: it provides a reusable expert-annotated benchmark for a domain where evaluation is notoriously difficult, gives the first systematic comparison of LLM judges for HOBRE, and demonstrates a cost-aware routing mechanism. The release of the benchmark and code at the stated DOI supports reproducibility, and the empirical breadth (9 judge LLMs, 81 configurations, 3 tasks) is substantial. The analysis of configuration variability and the cost–correlation trade-offs is useful for practitioners. However, the gold-standard labels in BinJudgeBench are not reference-free in the sense the paper claims, and the headline comparison in Table 3 is affected by test-set configuration selection and lacks statistical inference. These issues bear directly on the validity of the main quantitative claims.","major_comments":[{"comment":"The benchmark is described as 'reference-free,' but the annotation protocol explicitly provides annotators with the original source code S_f and requires them to judge whether the output is 'closer to a higher-level representation of the source code S_f' (Table 1, dimension 2). Thus the gold labels measure source-informed utility, not the judgment of an analyst who, as the paper's own motivation states (Section 1), lacks source access. Every correlation in Tables 3 and 7 is therefore computed against a target construct that differs from the reference-free setting the paper claims to evaluate. The independent industrial expert check in Section 6.2 uses the same source-visible protocol and does not remedy this gap. The authors should either re-annotate a subset of samples without source access, or revise the benchmark's claims to 'source-informed human judgment' and temper the 'reference-free' terminology throughout.","section":"Section 3.3.2 / Table 1"},{"comment":"The comparison between LLM-as-a-Judge and traditional metrics is not apples-to-apples. Section 4.2 states that Table 3 'reports only the configuration (i.e., prompting strategy and decoding temperature) that achieves the highest average correlation for each LLM across the three tasks,' so each LLM's reported correlation is selected post-hoc on the very human labels used for evaluation. In contrast, the traditional-metric baseline is the unweighted average over 11 metrics (the 'Average' row), with no per-metric best-selection or tuning. Consequently, the 63.20% versus 35.04% gap conflates the effect of the evaluation paradigm with the effect of test-set configuration selection. A fairer comparison would either average LLM performance over all 81 configurations, report the best traditional metric per task, or use a validation split for configuration selection.","section":"Section 4.2 / Table 3"},{"comment":"The paper repeatedly claims that LLM-as-a-Judge 'substantially outperforms' traditional metrics and 'significantly outperforms' them, yet no confidence intervals, significance tests, or effect-size measures are reported for the key correlation differences. With n = 1,233 samples, the reported gaps between LLM judges (for example, Codestral at 53.30% versus DeepSeek-V3.2 at 70.87% in Table 3) may or may not be statistically reliable; the same applies to the LLM-versus-traditional comparison. The authors should report bootstrap confidence intervals for Kendall's tau and apply a paired test suitable for dependent correlations (e.g., Williams's test or a bootstrap paired difference test) for the central comparisons.","section":"Section 4.2 / Abstract"},{"comment":"The 'oracle gap' analysis defines the Oracle Configuration as selecting, for each sample, the configuration that minimizes squared error with the human score. This oracle is, by construction, the best achievable on the test set and cannot be approximated without access to the labels; its large gap relative to static configurations (e.g., 70.31% vs 92.95% τ for FNR) therefore partly reflects the oracle's use of the gold labels rather than intrinsic sample heterogeneity. The 'no one-size-fits-all' conclusion should be supported primarily by the cross-validated gains of BinJudge in Table 7, which are meaningful, rather than by the raw oracle gap, which is an unreachable upper bound. The current presentation overstates the strength of the negative result from the oracle comparison alone.","section":"Section 4.4 / Table 5"}],"minor_comments":[{"comment":"The text says 'totaling 11 traditional automated evaluation metrics,' but Table 3 lists 12 metric rows (F1-score plus BLEU, METEOR, ROUGE-L, ChrF++, CrystalBLEU, CodeBLEU, RUBY, SentenceBERT, MoverScore, BERTScore, and CodeBERTScore). Please verify the count and adjust the text or table accordingly.","section":"Section 4.1.2 / Table 3"},{"comment":"The 'Average' row for traditional metrics is computed across tasks and metrics, but F1-score is reported only for FNR. The paper should state how the per-task averages handle missing values (e.g., whether the overall average is a simple mean of all reported cells).","section":"Table 3"},{"comment":"The utility function introduces hyperparameters λ = 0.1 and Softmax temperature T = 0.5 without any sensitivity analysis. Since BinJudge's cost–correlation trade-off directly depends on these choices, a brief sensitivity study or at least a discussion of their stability would strengthen the paper.","section":"Section 5.1"},{"comment":"The sampling strategy first draws a random sample of 385 responses and then supplements with 26 targeted samples to cover all 192 (model, architecture, optimization level) triplets. This supplemental step deviates from a simple random sample; the authors should note how this affects the representativeness of BinJudgeBench relative to the underlying 2,772,768-response pool.","section":"Section 3.3.1"},{"comment":"The 'fluency trap' analysis relies on a manually selected sub-pool of 100 'seemingly fluent' samples from the unsupported pool. The selection criteria are described narratively; providing a reproducible definition or the actual selection code would allow others to verify this finding.","section":"Section 4.2"},{"comment":"The terms 'reference-free' and 'reference-free evaluation benchmark' are used prominently, but per the construct-validity issue raised above, the benchmark's annotations are source-informed. At minimum, the abstract and conclusion should qualify these claims or the benchmark should be re-annotated without source access to justify the terminology.","section":"Title and Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a camera-ready conference paper (ASE '26) with an acceptance date already listed. For a journal version, the construct-validity issue around the 'reference-free' labels is the most serious concern; the authors themselves provide the source code to annotators (Table 1), which contradicts the central framing. The lack of statistical inference and the test-set configuration selection in Table 3 are also load-bearing for the headline claim. If these are addressed with a re-annotated source-free subset or a reframed claim, plus proper significance testing and a fair baseline comparison, the paper could be suitable for publication. The authors have many self-citations, but that is not unusual in this area and not a concern by itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is the first serious attempt to use LLM-as-a-Judge for binary reverse engineering evaluation, and the benchmark and router are worth having. But read the headline numbers with care: the 63.20% vs 35.04% gap is partly a selection artifact, and the \"reference-free\" framing doesn't match how the labels were made.\n\nWhat's actually new: BinJudgeBench, 1,233 expert-scored samples spanning function name recovery, binary summarization, and decompilation optimization, with coverage across 6 architectures and 4 optimization levels. The paper also runs a broad configuration sweep (9 LLMs × 3 prompts × 3 temperatures) and shows that the best configuration varies by task and by sample. The BinJudge router, which picks a configuration per sample from a cheap feature extractor, is a practical contribution: it improves correlation over static configs and cuts API cost substantially. The data and code are released.\n\nThe study is well executed in its own terms. The annotation protocol includes calibration, independent scoring, agreement checks, and review meetings; the limits are discussed honestly in Section 6.2. The cost analysis is refreshingly concrete.\n\nThe soft spots are real, though. First, the gold standard is not reference-free. In Table 1 annotators are given the source code S_f and asked whether the output is \"closer to a higher-level representation of the source code.\" So the labels measure source-informed utility, not the judgment of an analyst working from a stripped binary alone. The paper's central claim—that LLM judges work for reference-free HOBRE evaluation—is measured against a target that had the reference. The independent industrial check used the same protocol, so it doesn't fix this. This doesn't kill the benchmark; it just changes what it validates.\n\nSecond, the headline comparison is apples-to-oranges. Each LLM's correlation is reported at its best prompt/temperature selected on the same human labels, while the traditional metric baseline is the average over 11 metrics without selection. No confidence intervals or significance tests are given. The gap would shrink, possibly a lot, if both sides were treated the same way.\n\nThird, the router's utility function uses the human labels to pick the per-sample best configuration, so the oracle and the training target are fitted to the same data. The 5-fold CV helps, but the reported gains are still bounded by the label quality.\n\nBottom line: a useful benchmark and a practical router for the HOBRE evaluation community. The central direction—LLM judges beat text-matching metrics here—is probably right. But the reference-free framing needs to be fixed, and the comparisons need to be fairer before the quantitative claims are taken at face value. I'd send it to peer review with major revision, and I'd bring it to the reading group.\n\nRecommendation: engage seriously.","headline":"A useful HOBRE LLM-judge benchmark and router, but the reference-free framing and headline comparison are both weaker than they look.","tokens_in":27836,"tokens_out":3371,"would_cite":true,"duration_ms":28582,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM judges can grade decompiled-code outputs without source references, reaching 63.20% average correlation with human experts versus 35.04% for traditional metrics, and per-sample configuration routing raises this by 4.5%-24.7% at lower…","keywords":["binary reverse engineering","LLM-as-a-Judge","reference-free evaluation","function name recovery","binary code summarization","decompilation optimization","human judgment correlation","BinJudgeBench"],"falsifier":"Re-annotate a random subset of BinJudgeBench with independent reverse-engineering professionals who see only stripped pseudocode and no source code, and recompute Kendall correlations; if LLM judges drop to near the 35% level of traditional metrics, or if BinJudge's routed configurations no longer beat static best configurations, the central claim fails. A second check is to audit the 149 hard samples where human and LLM scores diverge and see whether they share a feature that would flip the router's preferred configuration to a worse one.","tokens_in":26849,"feed_emoji":"⚙️","tokens_out":9359,"duration_ms":78233,"temperature":0.7,"pith_summary":"Human-oriented binary reverse engineering turns stripped decompiled pseudocode into readable function names, summaries, and source-like code, but there is no cheap way to grade such outputs when source code and executable tests are unavailable. This paper tries to establish that LLM-as-a-Judge, where a large language model scores an output directly against the pseudocode instead of matching it to a reference string, can stand in for human experts in that reference-free setting. On a new expert-annotated benchmark called BinJudgeBench covering three reverse-engineering tasks, the paper reports average correlation with human judgment of 63.20% for LLM judges, nearly double the 35.04% average of eleven traditional match- and embedding-based metrics. It also reports that no single judge configuration is best for every task or sample, and that a lightweight router, BinJudge, which picks a configuration per sample, improves correlation with human experts by 4.5%-24.7% while cutting API cost to 0.06x-0.84x of static best configurations.","feed_headline":"LLM judges beat text matching for binary reverse-engineering review","feed_subtitle":"Reference-free scoring aligns with human experts at 63%, nearly double the 35% of standard metrics.","key_machinery":"The load-bearing object is the judge configuration space plus a lightweight routing policy. From 9 backbone LLMs, 3 prompting strategies (zero-shot, few-shot, chain-of-thought), and 3 decoding temperatures, the paper forms 81 judge configurations, each measured by its correlation with expert scores and its API cost. BinJudgeBench supplies the gold standard: 1,233 samples across function name recovery, binary code summarization, and decompilation optimization, scored on a 1-5 Likert scale by three reverse-engineering experts using a two-tier rubric (semantic correctness and analyst utility shared across tasks, plus one task-specific dimension). BinJudge is the router: a frozen UniXcoder encoder embeds the stripped pseudocode, a learned task embedding adds task awareness, and a multilayer perceptron outputs a preference distribution over configurations, trained by KL divergence against a temperature-softmaxed target distribution derived from a sample-level utility $U_{i,c}=-(MSE_{i,c}+\\lambda\\cdot Cost_{i,c})$; the cost term is log-scaled so the router optimizes correlation and expense together.","core_discovery":"The central claim is that reliable HOBRE evaluation does not require source-code references at all: LLM judges can infer semantic correctness, analyst utility, and task-specific quality directly from the stripped pseudocode. The paper's measurements, on 1,233 expert-annotated samples, show the best LLM judges reach around 69% Kendall correlation with experts on function name recovery and 68% on binary code summarization, while eleven existing metrics average 35.04% across tasks. A configuration analysis over 81 judge setups shows rank flips between tasks and between samples, so the paper concludes that a one-size-fits-all judge does not exist. BinJudge, the proposed solution, learns to route each sample to a configuration by maximizing a human-utility objective that balances score accuracy with API cost; in five-fold cross-validation it beats every static configuration, improves Kendall correlation by 4.5%-24.7%, and spends 0.06x-0.84x the cost of static best configurations, though it still lands short of the oracle upper bound.","pith_inferences":["Editorial extension: if LLM judges genuinely track expert judgment, the same utility signal could be used as a training reward for HOBRE models, turning the evaluator into an optimizer; the paper only demonstrates post-hoc scoring.","Editorial extension: because BinJudge's inputs are just pseudocode and task type, the routing idea should transfer to other judge-design spaces, but that transfer is untested.","Editorial extension: the paper's own fluency-trap analysis suggests reference-free judging over-credits fluent but unsupported outputs, especially for function names; a judge that must cite a semantic anchor before scoring might reduce this bias.","Editorial extension: since BinJudgeBench annotators could see the original source code, a source-blind re-annotation study would test whether the benchmark's gold standard matches what real analysts, who lack source, would consider good output."],"forward_implications":["Researchers can grade function names, summaries, and optimized pseudocode without original source code or runnable test cases, the exact situation faced in malware and closed-source-firmware analysis.","Evaluation design should treat judge configuration as a tunable component: a single best static configuration leaves a measurable oracle gap that per-sample routing narrows.","Large-scale automated evaluation of HOBRE models becomes affordable: BinJudge's routed judging spends 0.06x-0.84x of the API cost of static best configurations.","BinJudgeBench provides reusable expert labels and a multi-dimensional rubric for future HOBRE model comparisons, independent of reference text."],"supporting_citations":[{"why":"Defines the LLM-as-a-Judge paradigm that the paper applies to HOBRE evaluation.","marker":"[27]"},{"why":"Establishes that LLMs have some binary code comprehension, the capability premise for using them as HOBRE judges.","marker":"[48]"},{"why":"Supplies LLM4Decompile, a response-generation model for decompilation optimization, and the test-based evaluation the paper argues is limited.","marker":"[54]"},{"why":"Supplies SymLM, a function name recovery model whose outputs are judged in BinJudgeBench.","marker":"[23]"},{"why":"Supplies BinT5, a binary code summarization model whose outputs are judged in BinJudgeBench.","marker":"[4]"},{"why":"Supplies ReCopilot, the binary-domain LLM used as a response generator and as evidence of binary-adapted models.","marker":"[8]"},{"why":"Supplies the correlation-coefficient methodology and the human-aligned readability assessment approach the empirical study adopts.","marker":"[38]"},{"why":"Precedent showing LLM-as-a-judge narrows the gap with human evaluation in software engineering, the claim this paper extends to HOBRE.","marker":"[71]"}],"fun_headline_variants":["LLM judges match humans on binary reverse engineering without references","Reference-free LLM evaluation beats code-based metrics on binary RE","BinJudge: adaptive judge selection boosts human correlation by up to 24.7%","LLM judges need no source code: 63% agreement with experts on binary RE","Routing LLM judges per sample cuts cost to 6% while improving accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gold-standard labels in BinJudgeBench come from three author-experts scoring 1,233 samples with the original source code available for reference; if those labels are noisy, biased, or unrepresentative of real reverse analysts working without source, every correlation and every router improvement is measured against a flawed target.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges match humans on binary reverse engineering without references","Reference-free LLM evaluation beats code-based metrics on binary RE","BinJudge: adaptive judge selection boosts human correlation by up to 24.7%","LLM judges need no source code: 63% agreement with experts on binary RE","Routing LLM judges per sample cuts cost to 6% while improving accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3567,"prompt_tokens":1095,"completion_tokens":2472,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":711,"completion_tokens_details":{"reasoning_tokens":2374}},"tokens_in":711,"tokens_out":2472,"duration_ms":17146,"temperature":1.0,"reasoning_tokens":2374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:54:49.038064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of BinJudgeBench with independent reverse-engineering professionals who see only stripped pseudocode and no source code, and recompute Kendall correlations; if LLM judges drop to near the 35% level of traditional metrics, or if BinJudge's routed configurations no longer beat static best configurations, the central claim fails. A second check is to audit the 149 hard samples where human and LLM scores diverge and see whether they share a feature that would flip the router's preferred configuration to a worse one.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that LLMs have some binary code comprehension, the capability premise for using them as HOBRE judges."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies LLM4Decompile, a response-generation model for decompilation optimization, and the test-based evaluation the paper argues is limited."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Precedent showing LLM-as-a-judge narrows the gap with human evaluation in software engineering, the claim this paper extends to HOBRE."}],"review_version":1}