{"id":"a2e17cbf-4a9a-4c02-a590-02f1f5d9a9cf","arxiv_id":"2501.11001","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces ScaMaha, a Java static-analysis tool for parsing code into XML, computing metrics, and visualizing structure, with a claimed yet weakly verified 100% precision and recall on five case studies.","lead":"ScaMaha is a new software tool that reads Java source code, converts it into an XML description, computes code metrics, and draws diagrams of classes, methods, and inheritance. The paper reports perfect accuracy on five test programs, but the evaluation relies on the author's own documentation and manual review rather than an independent benchmark.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ScaMaha's 100% completeness claim is circular: the meta-model defines which identifiers and relations count, so any Java construct outside that entity set is invisible to both extraction and manual ground truth.","rationale":"The reader identified the self-defined ground truth as the weakest assumption, which is closely related to but not identical with the concern raised here. The reader's concern is that the reference (documentation plus manual review) may be incomplete or erroneous, so 100% precision/recall could be an artifact of a flawed gold standard. My concern goes one step deeper: the ground truth is not merely potentially flawed, it is definitionally aligned with the tool's meta-model. The meta-model enumerates the only entity kinds the parser can emit, and the reference was constructed by reviewing the same kinds of entities. Thus the 100% figure cannot distinguish complete extraction from extraction that is complete only for the whitelisted constructs. This is a more fundamental circularity than 'manual review may have missed something' because even a perfect manual review of the Java source would not include omitted categories unless the reviewer independently thought to enumerate anonymous classes, lambdas, generics, and annotations. I rate the reader's ground-truth concern as the symptom and the meta-model completeness definition as the cause. The concrete test proposed directly settles whether the guarantee extends beyond the meta-model. If the test shows omissions, the paper's headline guarantee is overbroad and should be conditioned on the meta-model scoping; if the test shows all constructs are captured, then the concern is resolved. I do not move the reader's verdict because the appropriate outcome remains conditional acceptance pending a reproducible evaluation that uses an independently constructed benchmark covering modern Java constructs and that fixes the Table III visualization-column inconsistency (all visualization columns are marked 'x' despite the text claiming generated and verified visualizations for every system). The paper has genuine strengths: it is an open-source tool, the pipeline is described concretely with XML listings, and the runtime numbers show scalability on ArgoUML. Machine-checked proofs are not present, but the reproducibility of the code artifacts is a real positive. The central claim, however, is not established by the current evaluation design.","tokens_in":22323,"tokens_out":3596,"duration_ms":43589,"concrete_test":"Construct a minimal Java project that deliberately includes constructs absent from the ScaMaha meta-model: one enum with constants, one anonymous inner class, one lambda expression, one generic class with a type parameter used in a field, one annotation type and one annotation use, and a static initializer block. Run the ScaMaha parser on this project and inspect the resulting code file and metrics file. If any of these constructs (or their names, constants, or relations, e.g. calls from the lambda or anonymous class methods) are missing from the output, the 'no identifier or relation will be lost' guarantee is false for standard Java. If all appear correctly, the claim still needs to be explicitly scoped to the meta-model entity set before being accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central guarantee (Section 4: 'ScaMaha guarantees that software developers will not lose any code identifier or relation from the software code') holds only inside the entity universe defined by the ScaMaha meta-model in Section 3 (Figure 6, Listing 1). That meta-model enumerates packages, classes, attributes, methods, parameters, comments, local variables, and the relations inheritance, attribute access, and method invocation. It does not mention enum declarations or enum constants, anonymous and local classes, lambda expressions, generic type parameters and arguments, annotation declarations/uses, static initializer blocks, or imported types. The parser is built to extract this fixed set, and the evaluation's ground truth is the author's manual review plus existing documentation, which look for the same meta-model entities. Therefore a 100% precision/recall result is obtained by construction: any construct outside the meta-model is systematically absent from both the tool output and the reference list. The claim of no loss is thus not an empirical finding about Java reverse engineering completeness; it is an assertion that the meta-model's chosen categories are exhaustive. If a real Java system contains any of the omitted constructs (ArgoUML and Rhino almost certainly do, e.g. anonymous classes and generics), those constructs' identifiers and relations would be dropped without affecting the reported metrics, because the reference solution never contains them either. This is the load-bearing weakness: the evaluation cannot confirm the guarantee, only the fidelity of extraction within a pre-defined whitelist.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ScaMaha, a Java reverse-engineering tool that parses object-oriented source code into an XML code file, computes software metrics from that file, and produces several visualizations (code organization, inheritance, method invocation, polymetric views, tag clouds). The tool is built around a fixed meta-model (Section 3, Figure 6, Listing 1) and uses Eclipse JDT/AST for parsing. It is evaluated on five Java case studies of increasing size, and the paper claims 100% precision, recall, and F-measure for all extracted code identifiers, relations, metrics, and visualizations (Section 4).","tokens_in":22620,"tokens_out":3469,"duration_ms":41717,"significance":"If the 100% accuracy and completeness claims were properly supported, ScaMaha would be a useful, easily extensible analysis and visualization platform, especially for small and medium Java systems. The paper's strengths are its concrete tool architecture, its public availability on GitHub and a companion page, and its use of five case studies of different sizes. However, the current evaluation does not establish the central claims: the ground truth is assembled from the author's own documentation and manual review, the meta-model explicitly omits whole classes of Java constructs, Table III is internally inconsistent about visualization results, and no baseline comparison is made. These issues affect the main contribution, so the manuscript needs substantial revision rather than minor polishing.","major_comments":[{"comment":"Table III marks every visualization column (Codea, Classb, Methodc, Polymetricd, Cloude) with '×' for all five case studies, while the text in Section 4 states that the table 'shows the mined code visualizations for each case study' and the paper claims 100% accuracy for visualizations. If '×' means the visualization was not generated, the text is contradicted; if it means the visualization was generated, the table provides no evidence of content or correctness. The symbol must be defined and the per-case-study visualization results must be reported consistently with Figures 8–14.","section":"§4, Table III"},{"comment":"The paper's central guarantee that 'ScaMaha guarantees that software developers will not lose any code identifier or relation from the software code' is internally inconsistent with the meta-model definition in Section 3, which explicitly says 'this model doesn't show all OO entities' and Listing 1 has no representation for enums, anonymous/local classes, lambdas, generic type parameters or arguments, annotations, static initializers, or imports. A parser built on this meta-model cannot lose what it does not represent, so the guarantee holds only inside the entity universe chosen by the authors, not for arbitrary Java code. The claim needs to be scoped to the meta-model's entity set or the meta-model must be expanded.","section":"§3 (Figure 6, Listing 1) and §4"},{"comment":"The precision/recall computation is circular. The paper states: 'the author uses two resources to evaluate the obtained results. The first is the available software documents, and the second is the manual review of software code.' Both resources are produced or selected by the same author and look for the same entities that the ScaMaha meta-model defines. Consequently, any construct outside the meta-model is absent from both the tool output and the reference list, so a 100% result is obtained by construction. An independent ground truth is needed, for example counts produced by Eclipse JDT's AST visitor, javap output, or another parser such as Spoon or SrcML, and the comparison should be reported per entity type.","section":"§4, evaluation methodology"},{"comment":"The reasoning for the 100% score is not an evaluation. The paper says 'the proposed approach returns 15 classes from the software code' and then asserts that 'the software code actually contains only 15 classes' and 'the retrieved value of each metric is totally correct,' but no independent reference count is given for classes, methods, invocation relations, attribute accesses, or LOC. The paper should provide, for each case study and each artifact, a table with reference values, extracted values, true positives, false positives, false negatives, and the resulting precision and recall, rather than a blanket 100% claim.","section":"§4, mobile photo evaluation"},{"comment":"No comparison is made with existing parsers or reverse-engineering tools, despite Section 2 citing Spoon, SrcML, VerveineJ, MoDisco, and Moose. A baseline comparison on the same case studies (e.g., comparing the number of classes, methods, and relations extracted by ScaMaha with those extracted by Eclipse JDT itself or by Spoon) is necessary to support the claims of soundness and accuracy. Without such a comparison, the paper only demonstrates that the tool runs, not that it is correct or complete.","section":"§2 and §4"}],"minor_comments":[{"comment":"The table header uses 'Code' and 'Metrics file' with check marks, but no symbol legend is given; the column symbols should be made uniform and explained in the caption.","section":"§4, Table III"},{"comment":"Execution times are reported as single values without mentioning whether these are medians or averages over multiple runs, or the variance across runs; a sentence on measurement methodology would improve reproducibility.","section":"§4, execution times"},{"comment":"Listing 2 is a partial example, but the surrounding text and Figure 6 do not explain why the 'MethodAssignments' section appears in the XML schema but is never populated or discussed in the metrics; the tool should state whether assignments are extracted or reserved for future work.","section":"§3, Listing 2"},{"comment":"This paragraph appears inside the experimentation section but contains no experimental content; moving it to Section 3 or the conclusion would improve readability.","section":"§4, 'ScaMaha tool in a nutshell'"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the tool itself appears to be a genuine, usable artifact with public materials, but the paper's evaluation section needs substantial work before the 100% accuracy and completeness claims can be accepted. The internal contradiction in Table III and the circularity of the ground truth are the main concerns; I do not think the paper is intentionally misleading, but the claims are currently stronger than the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a tool paper with a real, usable artifact, but the headline result — 100% precision/recall on all artifacts — is a methodological artifact, not an empirical finding.\n\nWhat's actually new: ScaMaha is a fairly clean integration of Eclipse JDT AST parsing, an XML code model, conventional metrics, and Graphviz-based visualizations. The code file format is simple, the tool is open-source, and the author clearly knows the landscape: Moose/Famix, MoDisco, Spoon, SrcML, and CodeCity are all cited. The claimed novelty is the specific integrated pipeline and the XML model, which is a subset of standard metamodels.\n\nSoft spots: the evaluation is circular. The meta-model in Figure 6 and Listing 1 defines the entities (packages, classes, methods, attributes, parameters, local variables, comments) and relations (inheritance, attribute access, method invocation). The parser extracts exactly that set, and the ground truth is the author's own documentation and manual review, which look for the same entities. Any Java construct outside that universe — enums, lambdas, anonymous classes, generics, annotations, static initializers — is invisible to both the tool and the reference list. ArgoUML and Rhino almost certainly contain such constructs, so the guarantee that developers won't lose any identifier or relation holds only within a whitelist the author chose. That is load-bearing, not cosmetic. Second, Table III lists every visualization column as “×” for all five case studies, contradicting the text and figures that show visualizations. That table needs an honest correction before anything else. Third, there is no baseline comparison with Spoon or Moose, so “performance” is just raw execution times. To be fair, the paper does openly state that accuracy is measured against documentation and manual review, so a careful reader can see the weakness without digging.\n\nWho this is for: someone building a lightweight reverse engineering pipeline for small Java systems might find parts of this useful. But the evaluation as written would not survive a serious referee.\n\nRecommendation: yes, send it to peer review. A good referee can force a corrected Table III and a reworked evaluation with an independent ground truth (e.g., Spoon's model or a hand-built oracle) and honest reporting of what the meta-model omits. That revised version could be a decent tool paper. As it stands, the central accuracy claim is unsupported.","headline":"ScaMaha is a genuine integrated parser/analyzer/visualizer, but its 100% accuracy claim is an artifact of a self-defined entity universe rather than an empirical result.","tokens_in":23113,"tokens_out":3035,"would_cite":false,"duration_ms":31847,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ScaMaha, a static-analysis tool for Java object-oriented code, recovers every code identifier and relation, and that all evaluation metrics reach 100% on the five systems tested.","keywords":["reverse engineering","static code analysis","object-oriented software","source code parsing","software visualization","software metrics","Abstract Syntax Tree","ScaMaha"],"falsifier":"Run ScaMaha on a Java codebase whose known counts are generated independently by a compiler's symbol table, then compare the metrics file: if the reported number of classes, methods, inheritance relations, invocations, or accesses differs from the independent count, the 100% accuracy claim is false.","tokens_in":22148,"feed_emoji":"🔍","tokens_out":5773,"duration_ms":62510,"temperature":0.7,"pith_summary":"The paper sets out to show that one tool can perform the entire reverse-engineering chain for Java object-oriented code: parse source into an XML model, compute metrics from that model, and generate visualizations, without losing any identifier or relation. ScaMaha is the proposed tool, built around a dedicated meta-model and an Abstract Syntax Tree parser. If the reported evaluation is right, the tool achieves 100% precision, recall, and F-measure on all five case-study systems, making manual code review unnecessary for those extraction tasks. A sympathetic reader would take the central claim as: for the tested Java systems, every code artifact is recovered completely and exactly.","feed_headline":"ScaMaha reports 100% recall on every Java code artifact it extracts","feed_subtitle":"All code files, metrics, and visualizations matched documentation and manual review exactly in the paper's five case studies.","key_machinery":"The central object is the ScaMaha meta-model, an abstract description of object-oriented source code rendered as XML: packages, classes, attributes, methods, plus inheritance, method-invocation, and attribute-access relations. The machinery works as an interchange format: the parser maps Java source through the Abstract Syntax Tree into this XML code file, and both the analyzer and the visualizer consume only that XML, so the three components stay modular and extensible. The same meta-model is presented as language-independent, meaning another object-oriented language could feed it as long as a parser produces the same XML shape.","core_discovery":"On the paper's own terms, the discovery is that a static-analysis pipeline organized around a single XML schema can be complete and exact for object-oriented Java source. The parser walks the Abstract Syntax Tree and serializes packages, classes, interfaces, superclasses, attributes, methods, parameters, comments, local variables, exceptions, and the relations of inheritance, attribute access, and method invocation into a code file. The analyzer derives ten metrics, including counts of invocation and access relations, and the visualizer renders organization, inheritance, method-invocation, polymetric, and tag-cloud views from the same file. Across drawing shapes, mobile photo, health watcher, Rhino, and ArgoUML, the paper reports that every code file, metrics file, and visualization matched the reference obtained from documentation and manual review, yielding 100% on precision, recall, and F-measure. The paper states that this means software developers will not lose any code identifier or relation from the software code.","pith_inferences":["Editorial extension: a direct test the paper leaves implicit is to run the same pipeline on a system whose ground truth is generated by an independent compiler-grade AST tool rather than documentation and manual review; any discrepancy would narrow the 100% guarantee.","Editorial extension: because XML is an interchange format, the meta-model could serve as a neutral bridge between other analysis tools, but the paper only demonstrates the tool consuming its own XML, not interoperation with third-party parsers.","Editorial extension: the reported execution time of roughly 32 seconds for ArgoUML suggests interactive exploration of large systems is plausible, but the paper does not report memory use or behavior under repeated runs, so scalability beyond five systems remains open.","Editorial extension: a C++ parser emitting the same XML could extend the guarantee to a second language, but the completeness claim has only been shown for Java, so any cross-language claim would need its own evaluation."],"forward_implications":["Software developers can obtain a complete inventory of packages, classes, methods, attributes, comments, parameters, local variables, exceptions, and structural relations without running the code.","The XML code file becomes a reusable intermediate artifact that can feed other reverse-engineering tasks such as feature location, traceability recovery, summarization, and software-product-line analysis.","The metrics file gives a fast quantitative read on system size and complexity, as in the reported ArgoUML counts of 1,939 classes, 14,904 methods, and 56,758 invocation relations.","The visualizations offer an alternative path to understanding legacy code, including inheritance structure, invocation structure, package-level polymetric views, and identifier tag clouds.","A parser written for another object-oriented language could reuse the analyzer and visualizer unchanged, provided it emits the same XML format; the current version is limited to Java.","If the complete-extraction claim holds, manual code review for these particular identifiers and relations becomes replaceable by an automated, repeatable process."],"supporting_citations":[{"why":"Supplies the definitions of precision, recall, and F-measure that the paper uses to report 100% accuracy.","marker":"[80]"},{"why":"Provides the publicly available ScaMaha webpage holding the tool, tutorial, experimentation results, and case-study materials that back the reported artifacts.","marker":"[33]"},{"why":"Supplies the mobile photo software system used as the running example and as one of the five evaluated case studies.","marker":"[78]"},{"why":"Provides the Java implementation and documentation of the mobile photo software used as reference ground truth.","marker":"[79]"},{"why":"Provides the Rhino JavaScript-in-Java system, version 1.7R2, used as a medium-sized case study.","marker":"[82]"},{"why":"Provides the ArgoUML open-source system used as the large-scale case study with 271,690 lines of code.","marker":"[83]"},{"why":"Supplies the health watcher software, version 10, used as the web-based medium-sized case study.","marker":"[85]"}],"fun_headline_variants":["ScaMaha: 100% recall, 100% precision on Java source","ScaMaha nails every Java code element with exact metrics","ScaMaha extracts all Java code, zero loss, 100% recall","ScaMaha guarantees 100% capture of Java code artifacts","ScaMaha: exact and complete analysis of Java OO code"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100% scores depend on the assumption that the software documentation plus the author's manual reading of the code is a complete and correct list of every identifier, relation, and metric; if that reference list misses or mislabels anything, the perfect precision and recall figures are not established.","fun_headline_variants_meta":{"raw":{"variants":["ScaMaha: 100% recall, 100% precision on Java source","ScaMaha nails every Java code element with exact metrics","ScaMaha extracts all Java code, zero loss, 100% recall","ScaMaha guarantees 100% capture of Java code artifacts","ScaMaha: exact and complete analysis of Java OO code"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00117,"raw_usage":{"total_tokens":4910,"prompt_tokens":1087,"completion_tokens":3823,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":3725}},"tokens_in":703,"tokens_out":3823,"duration_ms":27040,"temperature":1.0,"reasoning_tokens":3725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:43:45.396378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run ScaMaha on a Java codebase whose known counts are generated independently by a compiler's symbol table, then compare the metrics file: if the reported number of classes, methods, inheritance relations, invocations, or accesses differs from the independent count, the 100% accuracy claim is false.","supporting_citations":[{"cited_title":"BushraDBR: An automatic approach to re- trieving duplicate bug reports,","cited_arxiv_id":null,"evidence_quote":"Supplies the definitions of precision, recall, and F-measure that the paper uses to report 100% accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the publicly available ScaMaha webpage holding the tool, tutorial, experimentation results, and case-study materials that back the reported artifacts."},{"cited_title":"Components meet aspects: Assessing design stability of a software product line,","cited_arxiv_id":null,"evidence_quote":"Supplies the mobile photo software system used as the running example and as one of the five evaluated case studies."},{"cited_title":"Figueiredo","cited_arxiv_id":null,"evidence_quote":"Provides the Java implementation and documentation of the mobile photo software used as reference ground truth."},{"cited_title":"(2023) Rhino: JavaScript in Java - Version 1.7R2","cited_arxiv_id":null,"evidence_quote":"Provides the Rhino JavaScript-in-Java system, version 1.7R2, used as a medium-sized case study."},{"cited_title":"Open-source software product line extraction processes: the argouml-spl and phaser cases,","cited_arxiv_id":null,"evidence_quote":"Provides the ArgoUML open-source system used as the large-scale case study with 271,690 lines of code."},{"cited_title":"(2023) Health watcher software - Version 10","cited_arxiv_id":null,"evidence_quote":"Supplies the health watcher software, version 10, used as the web-based medium-sized case study."}],"review_version":1}