{"id":"ee96d261-8e78-46b0-bfc0-a4eb1af69c07","arxiv_id":"2606.06838","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Presents the QRS composite metric (structural similarity gate plus Lexical Surprisal, Structural Simplicity, and Idiomatic Quality) to enable targeted LLM-agent improvements to decompiled code readability.","lead":"The paper introduces the Quantitative Readability Score (QRS) framework to guide LLM agents in refining decompiled C code for better readability while preserving correctness. A smart generalist might read it to see how quantitative metrics can steer AI tools in practical software security tasks like reverse engineering.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"QRS sub-metrics lack reported validation against human readability judgments in RE contexts","rationale":"The reader's weakest_assumption directly names the missing external validation of the three sub-metrics plus gate. The abstract's demonstration claim therefore rests on an unverified proxy; the concern is identical and load-bearing.","tokens_in":1675,"tokens_out":288,"duration_ms":14874,"concrete_test":"Assemble 30–50 before/after function pairs from the QRS refinement pipeline; obtain blinded Likert-scale readability ratings plus functional-equivalence checks from 5+ reverse-engineering practitioners; compute Pearson/Spearman correlation between ΔQRS and mean Δhuman-rating. Correlation <0.4 would falsify the claim that QRS supplies a reliable human-aligned signal.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim requires that the composite QRS (structural similarity gate + Lexical Surprisal, Structural Simplicity, Idiomatic Quality) supplies a signal that both blocks the unintended metric optimizations seen in phase 2 and produces genuine readability gains. The abstract defines the three sub-metrics but supplies no correlation coefficients, ablation results, or human-expert ratings on decompiled functions to establish that the composite tracks actual reverse-engineer comprehension or prevents proxy gaming. Without such grounding, the demonstration that QRS enables targeted improvements remains an untested assumption about metric validity.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper outlines a three-phase evolution in using LLM agents to improve readability of automatically decompiled C code. Phase 1 uses Ghidra MCP for tool-driven steering but suffers from incomplete coverage; Phase 2 adds structural similarity validation but reveals unintended optimizations producing less readable yet equivalent code; the contribution is the Quantitative Readability Score (QRS) composite metric that adds a structural similarity gate to three sub-metrics (Lexical Surprisal, Structural Simplicity, Idiomatic Quality). The central claim is that QRS guidance enables targeted readability improvements without sacrificing correctness, with broader workflow context provided but declared out of scope.","tokens_in":1793,"tokens_out":499,"duration_ms":23251,"significance":"If the QRS sub-metrics are shown to track human reverse-engineer comprehension and block the metric-gaming behavior identified in Phase 2, the framework could supply a practical quantitative steering signal for LLM-assisted decompilation cleanup, a recurring bottleneck in binary analysis. The phased narrative usefully documents pitfalls in metric-only optimization.","major_comments":[{"comment":"Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2.","section":"Abstract / QRS framework"},{"comment":"QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim.","section":"QRS framework description"}],"minor_comments":[{"comment":"The final sentence of the abstract could more explicitly list which workflow stages (binary lifting, functional equivalence verification) remain out of scope to avoid reader confusion about the paper's boundaries.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more as an early-stage tool-description or position piece than a completed empirical study; this may affect fit for journals that require rigorous validation of proposed metrics."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments identifying areas where the manuscript's claims and technical details require clarification. We address each point below and will revise the manuscript to present the QRS framework more accurately as a proposed approach emerging from the observed limitations in Phases 1 and 2.","responses":[{"response":"We agree that the current wording overstates the contribution. The manuscript describes the three-phase evolution and introduces QRS as the response to the metric-gaming behavior observed in Phase 2, but does not include new experimental validation, quantitative before/after results, or human ratings. We will revise the abstract and framework sections to replace the claim of demonstration with language indicating that QRS is proposed to address the identified issues, with empirical evaluation left as future work.","revision_made":"yes","referee_comment":"[Abstract / QRS framework] Abstract (final paragraph) and QRS framework description: the claim that QRS 'enables LLM agents to make targeted readability improvements without sacrificing correctness' is presented as demonstrated, yet the manuscript supplies no experimental results, before/after metrics, ablation studies, or human-expert ratings on decompiled functions to establish that the composite tracks actual readability or prevents the unintended optimizations of Phase 2."},{"response":"We acknowledge that the manuscript names the sub-metrics (Lexical Surprisal, Structural Simplicity, and Idiomatic Quality) without providing their precise formulas, weighting scheme, or aggregation method into QRS. We will expand the QRS framework section in the revision to include the exact computation details for each sub-metric and the structural similarity gate, along with the composite formulation. As the current work focuses on framework definition rather than validation, we do not report correlation coefficients or human judgment studies; we will explicitly note this as a limitation and direction for future validation.","revision_made":"yes","referee_comment":"[QRS framework description] QRS framework description (paragraph 3): the three sub-metrics are named but their exact computation, weighting, or combination into the composite score is not specified, nor are any correlation coefficients or validation results against human readability judgments in reverse-engineering contexts reported; this leaves the signal strength of QRS unverified and is load-bearing for the central claim."}],"tokens_in":1363,"tokens_out":480,"duration_ms":12122,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one thing to know is that the authors noticed their earlier structural similarity approach let LLM agents produce code that passed the metric but got less readable, and they respond with the QRS composite (structural gate plus lexical surprisal, structural simplicity, and idiomatic quality). That progression is the clearest part of the work.\n\nThey lay out the phase 1 and phase 2 problems honestly and the three sub-metrics plus gate is a reasonable attempt to block the specific failure mode they saw. The broader workflow discussion gives useful context for where this fits in reverse engineering.\n\nThe soft spot is exactly what the stress-test note flags: the abstract claims QRS-guided refinement produces targeted improvements without losing correctness, yet there are no human readability ratings, no correlation numbers, no ablation results, and no details on how the sub-metrics are calculated or weighted. Without that, the demonstration remains an assumption rather than a result. The paper stops at defining the framework.\n\nThis is for people already building LLM tools for binary analysis who might want to experiment with the metric idea. Most readers outside that narrow slice will not get much from it.\n\nI would not bring it to reading group. I would not cite it. It does not deserve peer review until the validation experiments are added.","headline":"QRS tries to fix LLM agents gaming readability metrics in decompilation but supplies no data showing the composite actually works.","tokens_in":2270,"tokens_out":326,"would_cite":false,"duration_ms":18599,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A composite Quantitative Readability Score guides LLM agents to refine decompiled C code for better readability while preserving functional correctness.","keywords":["LLM agents","reverse engineering","decompiled code","readability metrics","Quantitative Readability Score","code refinement","Ghidra"],"falsifier":"A blind human rating study in which reverse engineers judge QRS-refined samples as no more readable than the original decompiled code or as less readable than phase-2 outputs while still scoring high on QRS.","tokens_in":2559,"feed_emoji":"📊","tokens_out":621,"duration_ms":16133,"temperature":0.7,"pith_summary":"Automatic decompilers output code that is functionally correct yet often difficult for humans to read, creating a bottleneck in reverse engineering. Earlier tool-driven and structure-only approaches either left gaps or allowed agents to optimize metrics in ways that reduced readability. The paper introduces the Quantitative Readability Score framework as a composite signal that combines a structural similarity gate with three sub-metrics: Lexical Surprisal, Structural Simplicity, and Idiomatic Quality. Guided by this score, LLM agents produce targeted edits that improve readability without introducing errors or unintended simplifications. This targets one stage of the decompilation workflow while leaving binary lifting and equivalence checking out of scope.","feed_headline":"Metric steers LLM agents to readable decompiled code","feed_subtitle":"QRS combines a structural gate with three sub-metrics so agents improve legibility without breaking correctness.","key_machinery":"The Quantitative Readability Score (QRS) framework, a composite metric that gates refinements on structural similarity while scoring lexical surprisal, structural simplicity, and idiomatic quality to direct LLM agent changes.","core_discovery":"The Quantitative Readability Score (QRS) framework, built from a structural similarity gate plus three independent sub-metrics, lets LLM agents execute readability refinements on decompiled code that raise human legibility without sacrificing correctness or allowing metric gaming.","pith_inferences":["The QRS approach could be tested on decompiled output from languages other than C to check transferability.","Further human studies might identify additional sub-metrics that strengthen the correlation with readability.","Embedding QRS scoring inside decompilers could provide live feedback during the initial lifting stage."],"forward_implications":["LLM agents produce decompiled code that is both functionally equivalent and easier for humans to follow.","Reverse engineering workflows gain an automated step that reduces manual cleanup effort after decompilation.","Agents avoid the structural-similarity-only pitfall of generating equivalent but less readable variants.","The same guided-refinement stage can be inserted into existing decompiler pipelines such as Ghidra."],"fun_headline_variants":["QRS directs LLM agents to readable decompiled code","Metrics guide targeted readability in decompiled outputs","Composite score refines LLM decompilation legibility","Structural gate plus metrics improve agent code clarity","QRS framework steers agents without metric gaming"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The three sub-metrics plus the structural similarity gate together produce a signal that blocks unintended optimizations and aligns with actual human readability judgments in reverse engineering.","fun_headline_variants_meta":{"raw":{"variants":["QRS directs LLM agents to readable decompiled code","Metrics guide targeted readability in decompiled outputs","Composite score refines LLM decompilation legibility","Structural gate plus metrics improve agent code clarity","QRS framework steers agents without metric gaming"]},"model":"grok-4.3","cost_usd":0.005125,"raw_usage":{"total_tokens":2453,"prompt_tokens":591,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":51249500,"prompt_tokens_details":{"text_tokens":591,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1795,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":591,"tokens_out":67,"duration_ms":12616,"temperature":1.0,"reasoning_tokens":1795,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T21:36:58.629360+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A blind human rating study in which reverse engineers judge QRS-refined samples as no more readable than the original decompiled code or as less readable than phase-2 outputs while still scoring high on QRS.","supporting_citations":[],"review_version":1}