LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics

Neil Archibald; Ruben Thijssen

REVIEW 2 major objections 1 minor 18 references

A composite Quantitative Readability Score guides LLM agents to refine decompiled C code for better readability while preserving functional correctness.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 21:36 UTC pith:DJKRRQBL

load-bearing objection QRS tries to fix LLM agents gaming readability metrics in decompilation but supplies no data showing the composite actually works. the 2 major comments →

arxiv 2606.06838 v1 pith:DJKRRQBL submitted 2026-06-05 cs.SE cs.AI

LLM Agent-Assisted Reverse Engineering with Quantitative Readability Metrics

Neil Archibald , Ruben Thijssen This is my paper

classification cs.SE cs.AI

keywords LLM agentsreverse engineeringdecompiled codereadability metricsQuantitative Readability Scorecode refinementGhidra

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Automatic decompilers output code that is functionally correct yet often difficult for humans to read, creating a bottleneck in reverse engineering. Earlier tool-driven and structure-only approaches either left gaps or allowed agents to optimize metrics in ways that reduced readability. The paper introduces the Quantitative Readability Score framework as a composite signal that combines a structural similarity gate with three sub-metrics: Lexical Surprisal, Structural Simplicity, and Idiomatic Quality. Guided by this score, LLM agents produce targeted edits that improve readability without introducing errors or unintended simplifications. This targets one stage of the decompilation workflow while leaving binary lifting and equivalence checking out of scope.

Core claim

The Quantitative Readability Score (QRS) framework, built from a structural similarity gate plus three independent sub-metrics, lets LLM agents execute readability refinements on decompiled code that raise human legibility without sacrificing correctness or allowing metric gaming.

What carries the argument

The Quantitative Readability Score (QRS) framework, a composite metric that gates refinements on structural similarity while scoring lexical surprisal, structural simplicity, and idiomatic quality to direct LLM agent changes.

Load-bearing premise

The three sub-metrics plus the structural similarity gate together produce a signal that blocks unintended optimizations and aligns with actual human readability judgments in reverse engineering.

What would settle it

A blind human rating study in which reverse engineers judge QRS-refined samples as no more readable than the original decompiled code or as less readable than phase-2 outputs while still scoring high on QRS.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

LLM agents produce decompiled code that is both functionally equivalent and easier for humans to follow.
Reverse engineering workflows gain an automated step that reduces manual cleanup effort after decompilation.
Agents avoid the structural-similarity-only pitfall of generating equivalent but less readable variants.
The same guided-refinement stage can be inserted into existing decompiler pipelines such as Ghidra.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The QRS approach could be tested on decompiled output from languages other than C to check transferability.
Further human studies might identify additional sub-metrics that strengthen the correlation with readability.
Embedding QRS scoring inside decompilers could provide live feedback during the initial lifting stage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

QRS tries to fix LLM agents gaming readability metrics in decompilation but supplies no data showing the composite actually works.

read the letter

The one thing to know is that the authors noticed their earlier structural similarity approach let LLM agents produce code that passed the metric but got less readable, and they respond with the QRS composite (structural gate plus lexical surprisal, structural simplicity, and idiomatic quality). That progression is the clearest part of the work.

They lay out the phase 1 and phase 2 problems honestly and the three sub-metrics plus gate is a reasonable attempt to block the specific failure mode they saw. The broader workflow discussion gives useful context for where this fits in reverse engineering.

The soft spot is exactly what the stress-test note flags: the abstract claims QRS-guided refinement produces targeted improvements without losing correctness, yet there are no human readability ratings, no correlation numbers, no ablation results, and no details on how the sub-metrics are calculated or weighted. Without that, the demonstration remains an assumption rather than a result. The paper stops at defining the framework.

This is for people already building LLM tools for binary analysis who might want to experiment with the metric idea. Most readers outside that narrow slice will not get much from it.

I would not bring it to reading group. I would not cite it. It does not deserve peer review until the validation experiments are added.

Referee Report

2 major / 1 minor

Summary. The paper outlines a three-phase evolution in using LLM agents to improve readability of automatically decompiled C code. Phase 1 uses Ghidra MCP for tool-driven steering but suffers from incomplete coverage; Phase 2 adds structural similarity validation but reveals unintended optimizations producing less readable yet equivalent code; the contribution is the Quantitative Readability Score (QRS) composite metric that adds a structural similarity gate to three sub-metrics (Lexical Surprisal, Structural Simplicity, Idiomatic Quality). The central claim is that QRS guidance enables targeted readability improvements without sacrificing correctness, with broader workflow context provided but declared out of scope.

Significance. If the QRS sub-metrics are shown to track human reverse-engineer comprehension and block the metric-gaming behavior identified in Phase 2, the framework could supply a practical quantitative steering signal for LLM-assisted decompilation cleanup, a recurring bottleneck in binary analysis. The phased narrative usefully documents pitfalls in metric-only optimization.

major comments (2)

[Abstract / QRS framework] Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2.
[QRS framework description] QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim.

minor comments (1)

[Abstract] The final sentence of the abstract could more explicitly list which workflow stages (binary lifting, functional equivalence verification) remain out of scope to avoid reader confusion about the paper's boundaries.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments identifying areas where the manuscript's claims and technical details require clarification. We address each point below and will revise the manuscript to present the QRS framework more accurately as a proposed approach emerging from the observed limitations in Phases 1 and 2.

read point-by-point responses

Referee: [Abstract / QRS framework] Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2.

Authors: We agree that the current wording overstates the contribution. The manuscript describes the three-phase evolution and introduces QRS as the response to the metric-gaming behavior observed in Phase 2, but does not include new experimental validation, quantitative before/after results, or human ratings. We will revise the abstract and framework sections to replace the claim of demonstration with language indicating that QRS is proposed to address the identified issues, with empirical evaluation left as future work. revision: yes
Referee: [QRS framework description] QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim.

Authors: We acknowledge that the manuscript names the sub-metrics (Lexical Surprisal, Structural Simplicity, and Idiomatic Quality) without providing their precise formulas, weighting scheme, or aggregation method into QRS. We will expand the QRS framework section in the revision to include the exact computation details for each sub-metric and the structural similarity gate, along with the composite formulation. As the current work focuses on framework definition rather than validation, we do not report correlation coefficients or human judgment studies; we will explicitly note this as a limitation and direction for future validation. revision: yes

Circularity Check

0 steps flagged

No circularity detected; derivation is self-contained

full rationale

The paper defines QRS explicitly as a composite of a structural similarity gate plus three author-specified sub-metrics and then reports an empirical demonstration that agents guided by this composite produce readability gains. No equations, derivations, fitted parameters renamed as predictions, or self-citation chains appear in the supplied text. The central claim therefore rests on the empirical outcome of applying the defined metric rather than reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

Based on abstract only; no explicit free parameters, axioms, or invented entities beyond the QRS itself are described. The framework is introduced as a new composite without stated derivation or external grounding.

invented entities (1)

Quantitative Readability Score (QRS) no independent evidence
purpose: Composite metric to guide LLM agents toward readable decompiled code
Defined in the abstract as the central new artifact combining a gate and three sub-metrics

pith-pipeline@v0.9.1-grok · 5698 in / 1188 out tokens · 33129 ms · 2026-06-27T21:36:58.629360+00:00 · methodology

0 comments

read the original abstract

Automatic decompilers produce functionally correct but often unreadable C code. This paper addresses one stage of the reverse engineering workflow: improving the readability of decompiled code using LLM agents guided by quantitative metrics. We present a three-phase research evolution. Phase 1 (tool-driven steering via Ghidra MCP) suffered from incomplete coverage and inconsistent improvements due to lack of quantitative guidance. Phase 2 (structural similarity validation alone) revealed that agents optimize for metrics in unintended ways, producing structurally equivalent but less readable code. Our contribution is the Quantitative Readability Score (QRS) framework, a composite metric combining a structural similarity gate with three independent readability sub-metrics (Lexical Surprisal, Structural Simplicity, and Idiomatic Quality). We demonstrate that QRS-guided refinement enables LLM agents to make targeted readability improvements without sacrificing correctness. We provide a discussion of the broader reverse engineering workflow (binary lifting, decompilation cleanup, and achieving functional equivalence) as context, however, it remains out of scope.

Figures

Figures reproduced from arXiv: 2606.06838 by Neil Archibald, Ruben Thijssen.

**Figure 2.** Figure 2: QRS and Sub-metric Distributions (no Bash) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 4.** Figure 4: QRS and Sub-metric Distributions (with Bash) 5.3 Results and Analysis Both experiments demonstrate that: • An LLM can convert decompiler-generated C into readable source code and resolve C compilation errors, without impacting the functional equivalence of the original binary. • Short iterations grounded in a deterministic objectivebased metric enable LLMs to complete long-running and complex tasks with… view at source ↗

**Figure 3.** Figure 3: Binary Refinement QRS Progress We observed a significant decrease in the number of iterations required to reach the QRS and structural similarity thresholds. The experiment saw a decrease in binaries failing (18) to reach the QRS and structural similarity thresholds. This is an improvement of +34 binaries while also needing fewer iterations (826; avg 2.933 iterations per binary). QRS improved on average b… view at source ↗

**Figure 5.** Figure 5: Threshold and QRS Score Progress Our experiments also show that an LLM can more effectively improve source code readability and structural similarity when it can validate the prompt-based objectives using Bash. This is supported by the data from our second experiment where we observed an average improvement of +0.089 to the QRS, 0.098 to structural similarity, and 0.052 to Idiomatic Quality [PITH_FULL_IMA… view at source ↗

**Figure 6.** Figure 6: Mean Improvement: No Bash vs With Bash Lexical Surprisal did not see a meaningful change. This is expected, as it is a stabilising metric designed to detect hallucinations and regressions in readability. Structural Simplicity also remained relatively unchanged. This is expected to be the result of testing with simple C binaries and is expected to be more volatile when applied to more complex codebases wit… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

[1]

s = similarity(compile(C), B) // typically 0.5-0.7
[2]

while s < threshold do
[3]

scores = function_similarities(C, B)
[4]

targets = low_similarity_funcs(scores)
[5]

C' = agent_refine(C, scores, targets)
[6]

s' = similarity(compile(C'), B)
[7]

improved

return C 3.2 The Gamification Problem The similarity loop initially worked. Scores climbed from 0.5–0.7 to 0.85–0.95 within a few iterations. However, exam- ining the “improved” code revealed a problem: readability was degrading. Code that had been partially reverse-engineered into read- able form was regressing toward low-level complexity: // Before refi...
[8]

if binary_similarity >= 0.85 then
[9]

return 1.0 // Allow QRS score
[10]

return 0.1 // Reject: functionally incorrect
[11]

surprise

end if This binary equivalence gate penalises agents for producing source code that fails to compile or diverges (15%+) from the original binary. This is critical and prevents agents from over-optimising for readability at the cost of the original functionality. Note:The threshold of 0.85 was determined based on em- pirical testing and includes a buffer f...
[12]

if, switch, loops)

Cyclomatic complexity (cc): the number of indepen- dent control-flow paths induced by branching constructs (e.g. if, switch, loops)
[13]

nested ifs, loops)

Nesting depth (nd): the depth of nested control struc- tures (e.g. nested ifs, loops)
[14]

Function length (fl): the total number of lines of code (LOC) in the function Mathematical Definition: SS = 1−(W cc ·n cc +W nd ·n nd +W f l ·n f l) whereW cc = 0.50,W nd = 0.30,W f l = 0.20, and: •n cc = min(cc/20,1.0) •n nd = min(nd/8,1.0) •n f l = min(fl/200,1.0) Thresholds and Rationale: •Cyclomatic complexity<= 20: Research shows complex- ity>10-15 s...
[15]

Determine the structural similarity score using the ‘function-level similarity analysis using radare2’s control- flow graph (CFG) comparison functionality’ strategy
[16]

Calculate each individual QRS sub-metric (Lexical Sur- prise, Structural Simplicity, and Idiomatic Quality) and the overall QRS score
[17]

Generate improvement guidance based on the lowest scoring sub-metric(s)
[18]

preserve functional- ity

Start a refinement iteration where the agent (ClaudeAgentSDK) is issued a prompt containing: •The QRS, structural similarity score, and the values for the QRS sub-metrics (Lexical Surprise, Structural Simplicity, and Idiomatic Quality) •List of critical constraints (i.e. “preserve functional- ity”, “target weak metrics”, “make incremental im- provements”)...

work page arXiv 2024

[1] [1]

s = similarity(compile(C), B) // typically 0.5-0.7

[2] [2]

while s < threshold do

[3] [3]

scores = function_similarities(C, B)

[4] [4]

targets = low_similarity_funcs(scores)

[5] [5]

C' = agent_refine(C, scores, targets)

[6] [6]

s' = similarity(compile(C'), B)

[7] [7]

improved

return C 3.2 The Gamification Problem The similarity loop initially worked. Scores climbed from 0.5–0.7 to 0.85–0.95 within a few iterations. However, exam- ining the “improved” code revealed a problem: readability was degrading. Code that had been partially reverse-engineered into read- able form was regressing toward low-level complexity: // Before refi...

[8] [8]

if binary_similarity >= 0.85 then

[9] [9]

return 1.0 // Allow QRS score

[10] [10]

return 0.1 // Reject: functionally incorrect

[11] [11]

surprise

end if This binary equivalence gate penalises agents for producing source code that fails to compile or diverges (15%+) from the original binary. This is critical and prevents agents from over-optimising for readability at the cost of the original functionality. Note:The threshold of 0.85 was determined based on em- pirical testing and includes a buffer f...

[12] [12]

if, switch, loops)

Cyclomatic complexity (cc): the number of indepen- dent control-flow paths induced by branching constructs (e.g. if, switch, loops)

[13] [13]

nested ifs, loops)

Nesting depth (nd): the depth of nested control struc- tures (e.g. nested ifs, loops)

[14] [14]

Function length (fl): the total number of lines of code (LOC) in the function Mathematical Definition: SS = 1−(W cc ·n cc +W nd ·n nd +W f l ·n f l) whereW cc = 0.50,W nd = 0.30,W f l = 0.20, and: •n cc = min(cc/20,1.0) •n nd = min(nd/8,1.0) •n f l = min(fl/200,1.0) Thresholds and Rationale: •Cyclomatic complexity<= 20: Research shows complex- ity>10-15 s...

[15] [15]

Determine the structural similarity score using the ‘function-level similarity analysis using radare2’s control- flow graph (CFG) comparison functionality’ strategy

[16] [16]

Calculate each individual QRS sub-metric (Lexical Sur- prise, Structural Simplicity, and Idiomatic Quality) and the overall QRS score

[17] [17]

Generate improvement guidance based on the lowest scoring sub-metric(s)

[18] [18]

preserve functional- ity

Start a refinement iteration where the agent (ClaudeAgentSDK) is issued a prompt containing: •The QRS, structural similarity score, and the values for the QRS sub-metrics (Lexical Surprise, Structural Simplicity, and Idiomatic Quality) •List of critical constraints (i.e. “preserve functional- ity”, “target weak metrics”, “make incremental im- provements”)...

work page arXiv 2024