{"id":"3d25cb25-90f4-4a08-90fe-e21af838ec17","arxiv_id":"2608.12228","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A locally deployable LLM pipeline with documentation-based ranking and iterative classification extracts domain models from Java code, achieving an F1 of 0.90 for classes but with evaluation caveats.","lead":"This paper builds a pipeline that uses a local open-source language model, LLaMA3.1-70B, to reverse engineer domain models from Java source code, ranking classes by similarity to project documentation and classifying them iteratively. The authors report high F1 scores on ten small academic projects, but the evaluation has several caveats that make the headline numbers optimistic.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attribute and association F1 are computed only over true-positive classes, so the headline 'overall performance' overstates end-to-end extraction quality.","rationale":"The most defensible reading of the paper is that it proposes an iterative, ranking-guided LLM pipeline and evaluates it on ten student projects with expert-curated descriptions and known reference models. For that within-dataset claim, the load-bearing issue is not the alignment of the documentation—the paper explicitly says the descriptions were curated and treats them as inputs, so the evaluation shows what the pipeline can do given a good description—but the way the reported F1 scores are computed. Section 4.2.2 conditions attribute and association evaluation on true-positive classes, which removes class-misclassification errors from those metrics. Because the RQ1 answer and abstract then present 'strong overall performance' with F1s of 0.94 and 0.85, the numbers can be read as end-to-end model quality when they are actually component-level conditional accuracy. This is testable directly from the replication package. The reader's weakest assumption (documentation alignment) is a valid external-validity threat, and the paper itself lists sparse documentation as a threat; however, it does not affect what the reported numbers mean on the current dataset. The Table 2/Table 3 association F1 inconsistency (0.85 vs macro-average ~0.77) is a concrete symptom that the aggregation has not been independently audited. I therefore recommend keeping the CONDITIONAL verdict, with the condition that the authors recompute and clearly label end-to-end metrics.","tokens_in":17888,"tokens_out":7715,"duration_ms":65368,"concrete_test":"Using the replication package's raw per-element predictions, recompute attribute and association F1 over the complete output model for each case: credit an attribute only if its containing class is a true-positive class and the attribute matches the reference, count attributes from false-positive classes as false positives and attributes from false-negative classes as false negatives, and apply the same treatment to associations involving misclassified classes. If macro-averaged attribute/association F1 falls materially below 0.94/0.85, the RQ1 answer should be revised to describe conditional component accuracy rather than overall extraction performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2.2 restricts attribute evaluation to attributes of true-positive classes and association evaluation to associations between true-positive classes. The Table 2 headline F1 values (attributes 0.94, associations 0.85) therefore measure component accuracy conditional on correct class identification, not the quality of the extracted domain model as a whole. A false-positive class contributes all of its attributes and associations to precision errors in the delivered model, and a false-negative class removes its attributes and associations from recall, yet neither is penalized in the reported attribute/association scores. Since class precision is only 0.84 and per-case association precision falls to 0.43-0.47 (Climbsafe, AssetPlus in Table 3), end-to-end model-level F1 can be substantially lower than the table suggests. The RQ1 answer and abstract present these numbers as 'strong overall performance' without the TP-only caveat. The paper discloses the conditional metric in Section 4.2.2, but the caveat is absent from the central claim. A separate aggregate inconsistency—Table 2 reports association F1=0.85, while macro-averaging the ten per-case association F1 values in Table 3 gives about 0.77—reinforces that the headline numbers need independent re-derivation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for extracting domain models from Java source code using locally deployable open-source LLMs (LLaMA3.1-70B). The pipeline first extracts an implementation-level class diagram, ranks classes by SBERT cosine similarity between class names and project documentation, then iteratively classifies classes as domain-specific or implementation detail, filters attributes, and infers associations among the retained classes. Evaluation on ten academic Java projects with known reference domain models reports macro-averaged F1-scores of 0.90 for classes, 0.94 for attributes, and 0.85 for associations, together with ablations over tokenization and ranking. The paper answers four research questions and claims high recall with strong overall performance while remaining executable on local hardware.","tokens_in":18078,"tokens_out":6021,"duration_ms":51605,"significance":"The contribution is timely and practically relevant: it shows that a compact open-source model can perform a reverse-engineering task that has previously required proprietary models, and it does so with a transparent multi-stage pipeline, an ablation study isolating the contribution of semantic ranking, and a replication package. The approach is not mathematically circular: no parameters are fitted, and the pipeline is a fixed sequence of structural heuristics and LLM classifications. The main caveats are that the headline attribute and association figures are conditional on true-positive class identification, that the aggregate association numbers in Table 2 are inconsistent with the per-case values in Table 3, and that the ranking heuristic is evaluated with expert-curated documentation aligned with the ground-truth models. These issues do not invalidate the approach, but they require reframing or additional experiments before the 'strong overall performance' claim is fully supported.","major_comments":[{"comment":"Table 2 reports association precision 0.78, recall 0.93, and F1 0.85, but macro-averaging the per-case association values in Table 3 gives precision 0.68 and F1 0.77. Since the per-case values in Table 3 are already averages over three runs, this is not a run-level artifact. The abstract and the RQ1 answer build on the Table 2 numbers; please report exactly how the aggregate was computed (micro vs macro, pooled over runs) and correct the discrepancy before publication.","section":"§4.2.2, Tables 2–3"},{"comment":"Attributes and associations are evaluated only on true-positive classes, as stated in §4.2.2. This makes the reported attribute F1 of 0.94 and association F1 of 0.85 conditional on class identification rather than end-to-end model quality: a false-positive class contributes its attributes and associations to errors in a delivered model, while a false-negative class removes them from the model, and neither effect is reflected in these numbers. With class precision at 0.84 and per-case association precision as low as 0.43–0.47 for Climbsafe and AssetPlus (Table 3, cases 3, 4, 6), the difference can be material. Please report model-level metrics that include the effect of class errors, and add the 'conditional on true-positive classes' caveat to the RQ1 answer and abstract.","section":"§4.2.2, §5.1, Abstract"},{"comment":"The ranking heuristic in step B is validated using documentation that was curated by domain experts and is therefore aligned with the reference domain model. This is a load-bearing premise for the reported F1-scores: if real-world READMEs are outdated, vague, or use vocabulary that differs from the modeler's conceptualization, the ranking and subsequent iterative classification are likely to degrade. The threats-to-validity subsection acknowledges the risk qualitatively but does not test it. Please add an experiment with unmodified or minimally processed READMEs or requirements specifications, or explicitly narrow the applicability claim to settings where an aligned, high-level domain description is available.","section":"§4.2.1, §7.1"}],"minor_comments":[{"comment":"The table header does not indicate that the association columns are computed only between true-positive classes; add a footnote so that readers do not compare association metrics directly with the end-to-end class metrics.","section":"Table 3"},{"comment":"There are several typographical spacing errors, for example 'CoolSuppliesFxmlView scores0.4888despite' in §3.2 and 'Expert-developedCoolSuppliesdomain model' in the Figure 1 caption; please fix these before the camera-ready version.","section":"§3.2, Figure 1"},{"comment":"The RQ4 answer that performance is 'consistent' across implementations of the same domain is based on only three pairs and no statistical or per-pair quantitative comparison; report the paired differences and, if possible, a paired test or effect-size measure.","section":"§5.4"},{"comment":"The final sentence of §3.5 states that the pipeline recovers associations without directionality or multiplicities; since a UML class diagram normally includes multiplicities, this limitation should also be stated in the RQ1 interpretation and in the abstract, not only in the future-work section.","section":"§3.5, §7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of MODELS and the underlying idea is worth pursuing. The main risk is that the headline numbers are conditional and internally inconsistent; both are fixable with re-analysis and reframing. I would not reject on the academic-dataset concern alone, but the authors should be pushed to provide model-level metrics and to be precise about the documentation assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Main thing you should know: this is a solid, clearly written MDE paper with one real empirical insight and one reporting problem that needs fixing before publication. The new bit is not the components—SBERT ranking, iterative LLM classification, and local open-source models are all established—but the specific combination for extracting domain models from Java code, plus an ablation showing the semantic ranking is load-bearing: remove it, F1 drops from 0.90 to 0.69. That is a genuinely useful design finding, and the replication package plus three-runs averaging is more than many papers in this area ship.\n\nThe pipeline is well described. Step-by-step from code diagram to ranked classes to iterative classification to attribute filtering and association inference. RQ4 (same domain, two independent implementations) is a nice robustness check absent from related work. The authors also do not oversell the LLM itself—they are explicit that the iterative context is the main contribution. I buy the central mechanism.\n\nSoft spots, in order of importance.\n\nFirst, the headline numbers overstate end-to-end quality. Attribute and association metrics are computed only for classes already correctly identified as true positives (Section 4.2.2). That is disclosed in the metrics section, but the abstract and RQ1 answer present 0.94/0.85 as 'overall performance' without the conditional caveat. Since class precision is 0.84, a false-positive class's attributes and associations never count against the attribute/association scores, and false negatives are not penalized. End-to-end model-level F1 is lower than the table suggests. This is not a fatal flaw—conditional evaluation is common—but the central claim needs the caveat.\n\nSecond, the aggregate numbers do not match the per-case table. Macro-averaging the ten association F1 values in Table 3 gives roughly 0.77, not 0.85; similar gap for precision (0.68 vs 0.78). The discrepancy may be due to averaging across three runs at case level then macro-averaging, but the paper does not explain it. The authors need to reconcile the tables or show the aggregation method.\n\nThird, the input documentation was curated by domain experts and matches the ground-truth vocabulary. The paper acknowledges this in Section 4.2.1 and threats to validity, and I do not think it makes the result circular—the ranking heuristic is a plausible real-world signal. But it does mean the strongest evaluation numbers likely degrade on stale or vague READMEs.\n\nMinor: 'modest hardware' is 4x RTX 3090. That is not modest by typical industrial standards, and the paper itself later calls it high-end. Fix the wording.\n\nBottom line: the paper deserves a serious referee. The soft spots are report-level, not design-level, and the ranking ablation is worth preserving. I would send it to review with a request to fix the conditional metrics, reconcile the tables, and soften the claims. For someone working on LLM-based reverse engineering or MDE tooling, it's worth reading.","headline":"A useful local-LLM pipeline with a strong ranking ablation, but headline numbers are conditional and the aggregate tables need reconciling.","tokens_in":18643,"tokens_out":3023,"would_cite":true,"duration_ms":25961,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A ranked, iterative prompt loop lets locally deployed open-source LLMs extract domain models from Java code with F1 scores of 0.90 for classes, 0.94 for attributes, and 0.85 for associations.","keywords":["domain model extraction","reverse engineering","large language models","open-source LLMs","class diagrams","semantic similarity ranking","context window","model-driven engineering"],"falsifier":"Run the same pipeline on a set of industrial Java projects whose existing READMEs are known to be stale or mismatched with the implemented domain, and compare class F1 against independently built reference models; if F1 stays near 0.90 despite the misaligned documentation, the ranking heuristic is not as dependent on curated input as this evaluation suggests.","tokens_in":17668,"feed_emoji":"🤖","tokens_out":9349,"duration_ms":72057,"temperature":0.7,"pith_summary":"The paper sets out to establish that domain models—the conceptual classes, attributes, and relationships of a software system—can be recovered from source code using only compact, locally deployable open-source language models, without sending code to proprietary services. Its pipeline converts the code into an implementation-level class diagram, ranks classes by semantic similarity to a short project description, and then iteratively prompts the language model to separate domain classes from implementation details, filter attributes, and infer associations while accumulating context. On ten Java projects with expert-curated reference models, the paper reports F1 scores of 0.90 for classes, 0.94 for attributes, and 0.85 for associations, with high recall throughout. If true, this gives privacy-sensitive organizations a way to obtain higher-level models of legacy or AI-generated code on their own hardware.","feed_headline":"Local LLMs reverse-engineer domain models from Java at F1 0.90","feed_subtitle":"Ordering classes by similarity to project docs lets a small local model beat context-window limits.","key_machinery":"The load-bearing mechanism is a ranked, iterative prompt loop. Classes are ordered by cosine similarity between their names and the project documentation, computed with a sentence-embedding model, then fed one at a time to the language model with already-accepted domain classes included in each prompt, giving the model a growing local 'domain memory' without full-system context. Transitive associations reconnect domain classes that were separated when implementation classes were removed, and a final pairwise step asks the model to infer missing associations between previously unconnected domain classes. This loop is what lets a small local model approximate global reasoning under a restricted context window.","core_discovery":"The paper's central claim is that a deliberately ordered, iterative prompting loop lets a small local model reproduce the human modeler's abstraction step: it can distinguish domain concepts from implementation scaffolding and rebuild meaningful relationships between them. The ranking stage is decisive—removing semantic-similarity ranking drops class F1 from 0.90 to 0.69, mostly through lost precision—while tokenizing class names into separate words hurts (F1 falls to 0.64), so the paper concludes that raw identifiers carry useful signal. The paper further claims the pipeline remains stable across different implementations of the same domain model, and that the recovered model can be linked back to the code artifacts it came from.","pith_inferences":["Real-world documentation is rarely curated to match a reference model, so the F1 numbers likely sit near the optimistic end; a benchmark with uncurated READMEs would quantify the drop.","A confidence threshold on each LLM classification could turn the fully automatic pipeline into an interactive one that asks humans only about uncertain elements, improving precision where it is weakest.","Because the pipeline needs only an implementation-level class diagram as input, the same design should transfer to other object-oriented languages if a counterpart to the diagram extractor exists; only Java is tested here.","The relatively low association precision (0.78) suggests spurious links are the main bottleneck, so adding evidence from method call sites or type usage may raise precision more than better prompts alone."],"forward_implications":["Privacy-sensitive organizations can extract conceptual models from legacy or AI-generated Java code entirely on their own hardware, without sending source to third-party providers.","The semantic-similarity ranking is a required component: removing it lowers class F1 from 0.90 to 0.69, so deployments should keep or strengthen the ordering step.","Tokenizing class names before classification is counterproductive; the default configuration that keeps raw identifiers is the better choice.","The same reference domain model implemented by different code bases yields comparable extraction results, suggesting the method captures domain semantics rather than surface implementation patterns.","The extracted model retains links to the underlying code, which supports traceability and downstream engineering tasks."],"supporting_citations":[{"why":"Establishes the earlier LLM-based reverse engineering of class diagrams and the abstraction-level gap this approach addresses.","marker":"[12]"},{"why":"Provides the RAG-based alternative for context-limited LLM extraction, contrasted here as operating at a lower abstraction level.","marker":"[8]"},{"why":"Documents that proprietary LLMs are approaching human-level domain diagram generation, which motivates the search for local alternatives.","marker":"[40]"},{"why":"Supports the multi-step iterative prompting strategy that this pipeline adopts.","marker":"[55]"},{"why":"Shows that decomposition into subtasks beats single-step prompting, a design assumption of the pipeline.","marker":"[46]"},{"why":"Supplies the earlier GPT-based domain modeling problem setup that this work extends from textual requirements to source code.","marker":"[14]"},{"why":"Provides the sentence-embedding technique used for semantic similarity ranking.","marker":"[18]"},{"why":"Contains the dataset, prompts, and scripts behind the reported F1 numbers.","marker":"[34]"}],"fun_headline_variants":["Local LLM loop orders code, hits F1 0.90 for domain models","Heuristics + local LLM iteration recover domain models at F1 0.90","Ranking code by similarity unlocks local LLM domain extraction","Without full context, local LLM still finds domain concepts at F1 0.90","Small local model beats context limits for domain model mining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the short project descriptions used to rank classes are aligned with the reference domain model, because those descriptions were curated by domain experts; in real projects, READMEs that are vague or out of date would likely degrade the ranking and the classifications that depend on it.","fun_headline_variants_meta":{"raw":{"variants":["Local LLM loop orders code, hits F1 0.90 for domain models","Heuristics + local LLM iteration recover domain models at F1 0.90","Ranking code by similarity unlocks local LLM domain extraction","Without full context, local LLM still finds domain concepts at F1 0.90","Small local model beats context limits for domain model mining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1376,"prompt_tokens":855,"completion_tokens":521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":422}},"tokens_in":471,"tokens_out":521,"duration_ms":4847,"temperature":1.0,"reasoning_tokens":422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:11:53.056603+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a set of industrial Java projects whose existing READMEs are known to be stale or mismatched with the implemented domain, and compare class F1 against independently built reference models; if F1 stays near 0.90 despite the misaligned documentation, the ranking heuristic is not as dependent on curated input as this evaluation suggests.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the earlier LLM-based reverse engineering of class diagrams and the abstraction-level gap this approach addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RAG-based alternative for context-limited LLM extraction, contrasted here as operating at a lower abstraction level."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that proprietary LLMs are approaching human-level domain diagram generation, which motivates the search for local alternatives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that decomposition into subtasks beats single-step prompting, a design assumption of the pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sentence-embedding technique used for semantic similarity ranking."},{"cited_title":"2026.Towards Automated Domain Model Extraction from Source Code using Heuristics and Open-Source LLMs (Replication Package)","cited_arxiv_id":null,"evidence_quote":"Contains the dataset, prompts, and scripts behind the reported F1 numbers."}],"review_version":1}