{"id":"fcdeb447-5238-46e8-8a98-9a8a5388aec8","arxiv_id":"2607.23774","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SemRef refines existing architecture-recovery outputs with LLMs and dependency analysis, reducing distance to ground truth by 17.72–43.35% RDP across five metrics on 90 recoveries.","lead":"SemRef uses LLMs plus code dependencies to automatically clean up imperfect software-architecture recovery results. On nine systems and ten recovery tools it cuts the remaining error to ground truth by roughly 18–43%, and also builds hierarchical module summaries.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"Primary gains are computed on an oracle-flattened hierarchy; matching module count to ground truth can itself lift all five metrics, so semantic improvement is not isolated.","rationale":"I agree with the reader that the key uncertainty is whether LLM-guided moves converge toward human ground truth rather than a differently wrong partition, and that nine labeled systems limit generalization. But the sharper load-bearing point is the evaluation protocol around Finding 1: the output hierarchy is flattened using ground-truth module count before the headline metrics are computed. Because the input tools and ground truths often differ greatly in cardinality, and because the chosen metrics are not cardinality-invariant, a nontrivial portion of the universal improvement could be produced by granularity normalization and oracle depth choice rather than semantic file placement. This does not make the result circular against external GTs, and the paper’s direction is supported by positive PP values, top/leaf analyses, ablations, cost scaling, and multi-model checks. Still, the exact “all 90, all five metrics” claim and the 43.35% MoJoFM figure should be conditioned on a blind flattening/same-cardinality rerun plus a granularity-only control before the gains are attributed to semantic refinement.","tokens_in":27652,"tokens_out":2965,"duration_ms":128552,"concrete_test":"Recompute Tables 2–3 using only GT-blind flattening: for each of the 90 recoveries choose the SemRef level whose module count is closest to that input SAR tool’s own flat module count (ties to shallower), never using GT; run a split/merge-only arm with all LLM semantic steps disabled as the granularity-control. Report per-row sign and bootstrap CIs. If any metric becomes non-positive for >10% of rows, or mean RDP falls to the granularity-only arm, the headline is mainly a depth/cardinality artifact.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim rests on Table 2/3, but §4.2.1 first converts SemRef’s hierarchical output to a flat partition by selecting the depth whose module count is most similar to the ground truth. That is an oracle granularity choice. Several input tools were run with preset k=50 or ARC concerns=100 while ground-truth module counts range from 11 to 216 (Table 1), and all five metrics are sensitive to partition cardinality/granularity: MoJoFM edit operations, ARI’s chance-corrected partition agreement, and c2c overlap at threshold 0.66 can all improve when cluster count is moved toward the ground-truth count even if many file assignments are merely differently wrong. SemRef’s preprocessing explicitly performs this correction (split oversized modules, merge singletons), and Finding 5 says split/merge/file-refine all contribute. Thus Finding 1—“every metric improves on all 90” and RDP 17.72–43.35%—conflates better semantic placement with cardinality correction plus GT-matched depth selection. Table 5’s top/leaf averages are a useful mitigation, but they are still averages over levels that can mismatch GT and do not demonstrate per-recovery improvement under a blind, same-cardinality comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper presents SemRef, a multi-stage framework that refines the output of software architecture recovery (SAR) tools by combining LLM semantic judgments with dependency analysis. The pipeline: (i) preprocessing that splits oversized modules and merges singletons; (ii) file-level refinement in which a normalized file–module dependency-strength measure (Eq. 1) flags potential misplacements that an LLM then verifies; (iii) hierarchy construction seeded by high-cohesion, non-overlapping reference modules (Eq. 2); and (iv) iterative top-down refinement of top-level modules, module placement, and leaf contents, with hierarchical re-summarization until fixpoint. The evaluation applies 10 SAR tool variants to 9 projects with published ground-truth architectures (90 recoveries, 25.7K–11.7M NLOC), measures five established metrics (MoJoFM, a2a, ARI, a2a_adj, c2c_cvg), and reports that SemRef improves every recovery on every metric, with Reduction in Distance to Perfect (RDP) of 17.72–43.35%. Supporting studies include one-way ANOVA over tool/project effects, a cumulative stage ablation, token-cost scaling (approximately linear in file count; ~$2.72 for Chromium), a five-LLM comparison, and a Bash case study. Code and data are publicly archived.","tokens_in":28009,"tokens_out":5776,"duration_ms":132534,"significance":"If the results hold, this is the first LLM-based SAR-result refinement framework, and the evaluation is unusually broad for this area: 90 recoveries across 10 tool variants and 9 labeled systems, five standard metrics, statistical testing, ablation, cost analysis, and a multi-LLM generalizability check. Specific strengths worth naming: the use of RDP (normalized gain) to handle a2a's well-known compressed range; reproducibility measures (temperature 0, fixed seed, versioned model IDs); a public artifact on Zenodo; and the Table 5 robustness check using top-level and leaf-level flattening. However, the headline numbers are computed under an oracle granularity selection (the hierarchy depth whose module count best matches ground truth), so the 17.72–43.35% figures are upper bounds; the blind-strategy gains in Table 5 are roughly half as large (11.8–35.1%). The practical contribution is therefore real but more modest than the abstract suggests, and the central quantitative claim needs reframing before the results can be relied upon.","major_comments":[{"comment":"The primary comparison flattens SemRef's hierarchy by selecting 'the depth at which the hierarchy has the most similar number of modules to the ground truth.' This is an oracle granularity choice: it uses the evaluation target to pick the representation being evaluated. Finding 1 ('improved the accuracy of all 90 input architectures, regardless of the metric') and the abstract's 17.72–43.35% RDP range are established only under this oracle. Table 5's top-level/leaf strategies are a useful mitigation but report averages only; they do not show that per-recovery improvement survives a blind flattening, and the gains drop to 11.8–35.1%. The authors should (a) report per-recovery results under at least one blind strategy (leaf partition or a fixed-depth rule), (b) lead the abstract/conclusion with the blind numbers, and (c) explicitly label the depth-matched numbers as oracle upper bounds.","section":"§4.2.1, Tables 2–3, Finding 1"},{"comment":"Semantic improvement is not isolated from cardinality correction. Several input tools were run with preset k=50 (and ARC concerns=100) while ground-truth module counts span 11–216 (Table 1), and SemRef's preprocessing explicitly moves cardinality toward a more GT-like range (split oversized modules, merge singletons). All five metrics are sensitive to partition cardinality (MoJoFM merge/split operation costs, ARI's chance correction, c2c overlap at threshold 0.66), and Finding 5 confirms split/merge contribute measurably. Consequently Table 3 conflates better file placement with granularity correction. A control is needed: either re-run the clustering-based input tools with k set to the ground-truth module count, or include an algorithmic-only split/merge baseline (e.g., dependency-based, no LLM), so the LLM's semantic contribution can be quantified separately.","section":"§3.1, §4.1.2, Finding 5"},{"comment":"The data-leakage mitigation is weak relative to its importance. Six of nine ground truths predate GPT-4o-mini's cutoff, and Bash, Hadoop, and Chromium are prominent systems whose module structure is plausibly memorized; instructing the LLM to 'avoid considering the project background' (§3.1.1) is not verifiable. The cross-project ANOVA null result (Table 4) is indirect evidence with low power at n=9 per group. A concrete strengthening: report per-project RDP split by pre- vs post-cutoff ground truths, and/or add one recently released, less prominent system with a freshly labeled ground truth. This is a correctness-risk concern, not a circularity claim: the current evidence does not distinguish learned architectural judgment from recall.","section":"§5.2 (Threats to Validity)"}],"minor_comments":[{"comment":"'Projects ranging from thousands to trillions of lines of code' contradicts Table 1, where the largest system is 11.7M NLOC. Presumably 'millions' was intended. Also 'gains ranges from' → 'gains range from'.","section":"Abstract"},{"comment":"The text reports RDP 'increases ranging from 28.18% to 43.32%' and a c2c_cvg 'increment of 16.79%', but Table 3 shows 29.39%, 43.35%, and 17.72%. Please reconcile; the discrepancy suggests the text was written against an earlier experiment run.","section":"§4.2.1 text vs. Table 3"},{"comment":"Clarify whether average RDP is the mean of per-recovery RDPs or RDP computed on mean scores. Per-recovery RDP is unstable when M(A) approaches 1 (a2a inputs average ~0.81, so denominators ~0.19); a few recoveries with M(A) very close to 1 could dominate the mean. A sensitivity note or median would help.","section":"§4.2.1, Eq. (4)"},{"comment":"The ablation is cumulative (each bar removes all stages after stage k), not leave-one-out. Statements like 'the iterative refinement stage contributes the most' should be framed as marginal contribution given all prior stages; a leave-one-out ablation would support the stronger reading.","section":"§4.3, Fig. 5"},{"comment":"Ten ANOVA tests are reported without multiple-comparison correction; the MoJoFM tool effect (p=0.0622) is borderline and worth acknowledging as such.","section":"Table 4"},{"comment":"The c2c_cvg similarity threshold of 0.66 (taken from [91]) is a free parameter; a brief sensitivity check over a small threshold grid would strengthen the c2c_cvg claims, which are already the weakest (RDP 17.72%).","section":"§4.1.5"},{"comment":"The comprehensibility benefit (module names and summaries) is supported only by the single Bash case study and one developer-description comparison. Either temper the claim or add a small human assessment of summary quality.","section":"§4.2.2 / §5.1"},{"comment":"Averaging Llama3.3 over successful projects only introduces survivorship bias (flagged in the footnote, but the table still invites direct comparison). Consider also reporting a failure-inclusive aggregate.","section":"Table 7"},{"comment":"Typo: 'this fattening strategy' should read 'this flattening strategy'. Also 'input all the modules together to LLM could caused biased understanding' (§3.3) needs grammatical revision.","section":"§4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The evaluation infrastructure is solid and the requested controls (blind-flattening per-recovery results, a cardinality-matched baseline) appear feasible with the data the authors already have, so I expect a revision can resolve my main concerns. Two ground-truth sources (HDC, HDF, Libxml2) come from the authors' own prior SARIF paper [91], which is fine but worth noting when weighing the independence of the benchmark. The artifact links are listed as 'Anonymous' although the submission is not anonymized — a minor consistency point for the camera-ready."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a practical refinement pipeline, not a new recovery algorithm from scratch. They take existing SAR outputs, fix granularity (split/merge), move files via dependency flags plus LLM checks, build a hierarchy from cohesive reference modules, then iterate top-down. On 10 tools × 9 GT projects they report gains on every run; RDP 18–43% depending on metric, MoJoFM edit distance down ~43% on average.\n\nWhat is actually new is the staged fusion—dependency strength formulas (Eqs. 1–3) as cheap filters, LLM as semantic verifier, plus hierarchy and module summaries as side products. Evaluation is broad for SAR: ablation by stage, cost linear in file count, multi-LLM check, ANOVA on tool/project, and SARIF+SemRef beating best baselines. Code/data promised. Related work is honest about prior clustering and the thinner LLM-for-architecture line.\n\nSoft spots, in proportion. The stress-test on oracle flattening is fair: primary tables pick the hierarchy depth whose module count best matches GT, and split/merge already push cardinality. That can inflate all five metrics without pure semantic wins. Table 5 (top-only / leaf-only) still shows positive RDP, so the claim does not collapse, but the headline “all 90, all metrics” should be read with that caveat and ideally with same-k or blind-cardinality controls. Nine human GTs is the field’s ceiling, not a secret; still thin for generalization. No user study on whether hierarchy+summaries cut architect effort. LLM variance and prompt brittleness are lightly handled (temp 0, seed 1; Llama fails two large projects).\n\nMath and citations look fine—standard MoJoFM/a2a/ARI plus their a2a_adj and RDP; no circular scoring against self-defined targets. For anyone working recovery, maintenance, or LLM-for-structure this is worth a careful read. I would send it to referees; expect them to demand clearer same-granularity reporting and release packaging, not a rewrite of the core idea.","headline":"Solid empirical SE paper: multi-stage LLM+dependency refinement consistently lifts 90 SAR recoveries, with a real but partly mitigated granularity confound in the headline numbers.","tokens_in":28244,"tokens_out":532,"would_cite":true,"duration_ms":20667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"SemRef uses LLMs plus dependency analysis to automatically refine recovered software architectures, cutting average edit distance to ground truth by about 43%.","keywords":["Software Architecture","Architecture Recovery","Large Language Model","Dependency Analysis","Module Clustering","Architecture Refinement","Hierarchical Summarization"],"falsifier":"Run the same ninety-input protocol on a new set of systems that have independently labeled ground-truth architectures never seen by the LLM; if MoJoFM RDP stays near zero or turns negative while dependency-only baselines still improve, the central claim fails.","tokens_in":28143,"feed_emoji":"🏗️","tokens_out":951,"duration_ms":28601,"temperature":0.7,"pith_summary":"Automatic architecture recovery tools cluster source files into modules, but their results are often coarse or wrong, so architects still spend heavy manual effort cleaning them up. This paper introduces SemRef, a refinement framework that takes any existing recovery result and improves it by combining structural dependency checks with an LLM’s ability to judge what files and modules actually mean. On nine projects with published ground-truth architectures and ten recovery tools (ninety inputs total), every refined result scored higher on five standard similarity metrics; the normalized gains ranged from roughly 18% to 43%, with MoJoFM edit distance to ground truth falling by 43% on average. Token cost scales roughly linearly with the number of files, and the method also yields a hierarchical module tree plus natural-language summaries that make the recovered architecture easier to read. The practical claim is that architects can start from any off-the-shelf recovery tool and obtain a measurably closer, more comprehensible architecture without hand-editing every misplacement.","feed_headline":"LLMs cut architecture-recovery error by 43 percent","feed_subtitle":"SemRef refines any existing recovery tool’s output with dependency checks and semantic judgment, at linear token cost.","key_machinery":"SemRef: a multi-stage pipeline that first resizes modules, then flags file misplacements via a normalized dependency-strength formula, verifies them with an LLM, builds a hierarchy from cohesive reference modules, and iteratively refines top-level, intermediate, and leaf modules top-down until summaries and structure stabilize.","core_discovery":"Across all ninety recovery results from ten tools on nine ground-truth projects, SemRef consistently raises accuracy on five metrics; measured by reduction in distance to perfect, the gains run from 17.72% (c2c_cvg) to 43.35% (MoJoFM), equivalent to cutting the number of file-move and module-merge operations needed to reach ground truth by 43% on average.","pith_inferences":["If the same dependency-plus-LLM loop works for recovery refinement, a lighter variant might continuously monitor pull requests and flag architectural drift before it accumulates.","Organizations that already own proprietary clustering tools can treat SemRef as a drop-in post-processor rather than replacing their internal pipelines.","The nine labeled systems are still a small sample; a public leaderboard of new ground-truth architectures would let the community stress-test whether the 43% MoJoFM gain generalizes.","Because leaf-module summaries are generated bottom-up, the same machinery could feed higher-level design-decision or onboarding documents with little extra prompting."],"forward_implications":["Any existing recovery tool can be wrapped by SemRef and produce higher-accuracy flat or hierarchical architectures without changing the tool itself.","Pairing the strongest current recovery tool (SARIF) with SemRef yields the best published results on the nine-project suite, beating every standalone baseline.","Token cost grows linearly with file count, so the method remains affordable even on multi-million-line systems such as Chromium.","Stronger underlying LLMs further raise the same RDP numbers, so future model improvements translate directly into better refined architectures.","The produced hierarchy and module summaries give maintainers readable documentation even when original design docs are missing."],"fun_headline_variants":["SemRef cuts architecture-recovery error up to 43%","LLMs plus dependencies lift recovery accuracy 17-43%","SemRef refines ten tools' architectures with 43% MoJoFM gain","LLM semantic checks shrink recovery distance by 43%","Automatic refinement raises architecture recovery on five metrics"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That the LLM’s judgments of semantic cohesion and correct module placement, guided by the paper’s dependency formulas and prompts, line up with human ground-truth boundaries often enough for iterative refinement to converge on a better architecture rather than a differently wrong one.","fun_headline_variants_meta":{"raw":{"variants":["SemRef cuts architecture-recovery error up to 43%","LLMs plus dependencies lift recovery accuracy 17-43%","SemRef refines ten tools' architectures with 43% MoJoFM gain","LLM semantic checks shrink recovery distance by 43%","Automatic refinement raises architecture recovery on five metrics"]},"model":"grok-4.5","effort":"low","cost_usd":0.004414,"raw_usage":{"total_tokens":1308,"prompt_tokens":804,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":44144000,"prompt_tokens_details":{"text_tokens":804,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":436,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":804,"tokens_out":68,"duration_ms":8589,"temperature":1.0,"reasoning_tokens":436,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T12:55:22.808185+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the same ninety-input protocol on a new set of systems that have independently labeled ground-truth architectures never seen by the LLM; if MoJoFM RDP stays near zero or turns negative while dependency-only baselines still improve, the central claim fails.","supporting_citations":[],"review_version":1}