{"id":"8048ff24-b75a-45d3-9ded-fc1b3a21987e","arxiv_id":"2412.08352","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 17 microservice applications, Code2DFD achieved the highest individual F1-score of 0.86 among nine static architecture recovery tools, with an in-sample four-tool combination reaching 0.91.","lead":"This paper identifies 13 static analysis tools that recover microservice architectures and runs 9 of them on 17 open-source Java Spring applications, scoring their precision, recall, and F1. The best single tool scored 0.86 F1, and the best in-sample combination of four tools scored 0.91.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"F1 ranking is computed on different per-tool ground-truth sets, so 'Code2DFD is best' is not yet supported; a common-subset comparison is needed.","rationale":"The reader's stated weakest assumption is about ground-truth completeness, especially the annotation-based endpoint ground truth. That is a real concern, but I find a more actionable and more load-bearing issue in the way F1 scores are computed: the per-tool GT sets differ substantially, so the 'best individual tool' claim compares non-comparable quantities. The reader's rationale does mention 'different ground-truth sizes across tools make the F1 ranking partly incomparable,' so there is partial agreement, but it is not the named weakest assumption. I am not raising this as evidence of misconduct or sloppiness: the paper is transparent, includes a replication package, and follows a registered protocol. The concern is strictly about whether the central quantitative claim is supported by the reported analysis. A common-subset recomputation is feasible from the existing data and would settle the issue. Because the reader already issued a conditional verdict and my concern reinforces the need for that condition rather than overturning it, I recommend the verdict remain unchanged.","tokens_in":30472,"tokens_out":7998,"duration_ms":88417,"concrete_test":"Use the replication package (Zenodo 14179613) to recompute precision, recall, and F1 for every tool on the maximal set of applications that all tools with a given characteristic in scope can analyze, and at minimum on pairwise common subsets (e.g., Code2DFD versus RAD for endpoints on the 13 applications RAD supports). If Code2DFD's overall F1 remains highest on the common subset, the headline ranking survives; if any ordering flips, the current F1 table is an artifact of unequal GT sets and the paper should be revised to report common-subset scores.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline ranking in RQ4 is not a controlled comparison because each tool's F1 is computed over a different set of ground-truth items. Table 5d shows Code2DFD with F1=0.86 on GT=727 from 17 applications, while RAD has F1=0.79 on GT=391 from 13 applications, MicroDepGraph has F1=0.74 on GT=355 from 10 applications, and Attack Graph Generator has F1=0.60 on GT=468 from 9 applications. If a tool runs on only the easier applications in the benchmark, its precision and recall are measured on a different universe of components, connections, and endpoints. The paper reports per-tool execution counts in Table 6 but never reports a common-subset analysis, so the observed ordering could be an artifact of app difficulty being confounded with tool coverage. The same issue affects the per-characteristic rankings, e.g., RAD's endpoint F1=0.79 on GT=132 versus Code2DFD's F1=0.66 on GT=160; restricting both to the 13 applications RAD can analyze could change the comparison. The central claim that Code2DFD is the best individual tool therefore needs a same-items comparison before it can be accepted as a robust result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a multivocal literature review that identifies 13 static analysis architecture recovery tools for microservice applications and an execution-based comparison of nine of these tools on the microSecEnD dataset, which contains 17 Java/Spring microservice applications. The tools are compared on their ability to extract components, connections, and endpoints, with precision, recall, and F1-score computed against a manually extended ground truth. The headline claim is that Code2DFD is the best individual tool (overall F1 = 0.86) and that a combination of Code2DFD, MicroDepGraph, RAD, and RAD-source reaches F1 = 0.91. The study is described as the first execution-based comparison of such tools and is registered at MSR'24, with a replication package provided.","tokens_in":30659,"tokens_out":10320,"duration_ms":97999,"significance":"If the comparison is valid, this is the first independent execution-based assessment of static architecture recovery tools for microservices, a genuine contribution to a field that has relied on self-reported results. The strengths are substantial: the methodology was peer-reviewed as a registered report, the literature review used two independent coders with reported Cohen's kappa agreement, the output quantification was performed by two authors with conflict resolution, and a replication package is available. The paper also carefully documents execution obstacles and the reasons tools were excluded, which is valuable for reproducibility. However, the validity of the headline ranking depends on a controlled comparison across a common set of ground-truth items, and this is not currently established.","major_comments":[{"comment":"The headline ranking in RQ4 is not a controlled comparison because each tool's F1 is computed over a different set of ground-truth items. For example, Code2DFD has F1 = 0.86 on GT = 727 from 17 applications, while RAD has F1 = 0.79 on GT = 391 from 13 applications and MicroDepGraph has F1 = 0.74 on GT = 355 from 10 applications. If tools run on different subsets of the dataset, the observed ordering may reflect application difficulty or dataset coverage rather than tool capability. The paper reports per-tool execution counts (Table 6) but does not provide a common-subset comparison. A same-items analysis, such as restricting all tools to the applications that every tool can analyze, or reporting per-application metrics, is needed to support the claim in the abstract that one tool is 'best-performing' and the answer to RQ4.","section":"Section 4.1, Table 5d, Table 6"},{"comment":"The endpoint ground truth is defined using the same Java annotations (@RequestMapping, @PutMapping, @GetMapping, @RepositoryRestResource) that several evaluated tools (RAD, RAD-source, Code2DFD, Prophet) use for endpoint detection. The paper states that identifying these annotations is 'sufficient' to create the ground truth, but no independent endpoint inventory is provided. This is partially circular: the ground truth is constructed from the tools' own detection mechanism, so endpoint recall is measured against a standard the tools effectively define. Endpoints implemented through other mechanisms (e.g., Spring WebFlux functional routing, JAX-RS, or gateway configurations) are excluded by construction. The claim that the annotation set is complete for these applications needs stronger support, or the endpoint results should be framed as agreement with an annotation-based definition rather than absolute endpoint extraction accuracy.","section":"Section 2.3"},{"comment":"The semantics of combining tools when not all tools ran on all applications is underspecified. The paper says that for OR-combinations, if only one tool produced results for an application, its results count directly, but the behavior for AND-combinations in the presence of missing tool outputs is not defined. Table 8 reports a single GT per combination without explaining whether the GT is the union, intersection, or some other aggregate of the per-tool application sets. There is also an apparent internal inconsistency: in Table 8a, AMOR has GT = 144, identical to AGG's GT, even though MDG analyzed 10 applications; if MDG analyzed any applications outside AGG's set, the union GT should be larger. The RQ5 headline result (F1 = 0.91) depends on these combination definitions, so the methodology needs to be clarified and the numbers reconciled.","section":"Section 2.6, Table 8"}],"minor_comments":[{"comment":"The text states that seven tools extract components and seven tools extract endpoints, but Table 4 shows eight tools for each (components: AGG, C2D, MDG, MMI, MTO, PRO, PR2; endpoints: AFA, C2D, MGR, PRO, PR2, RAD, RAS, CMA). The text and table should be aligned.","section":"Section 3.4, Table 4"},{"comment":"The sentence beginning 'A look into the individual tools’ performance – especially those that failed to produce meaningful results – shows,' is incomplete; it appears to be cut off and should be finished or removed.","section":"Section 5.1"},{"comment":"The manual quantification of true positives, false positives, and false negatives is reported as being performed by two authors independently, but no inter-rater agreement value is reported for this step, in contrast to the literature review phases. Reporting this agreement would strengthen the reliability of the core measurements.","section":"Section 2.5"},{"comment":"The conflict-of-interest statement declares no conflicts, but several evaluated tools (e.g., Code2DFD, microMiner) and the microSecEnD dataset are authored by members of this paper's author team. A disclosure and a brief discussion of any mitigation measures would be appropriate.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid candidate for the journal if the authors address the comparability issue. The registered report process and replication package are strong assets. However, the author team evaluates tools they have themselves developed, and the conflict-of-interest declaration should be revisited. The editor may also want to ensure that the common-subset analysis is a mandatory revision, since the headline ranking is the paper's central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the execution study: nine static architecture recovery tools run on a common benchmark, with a registered protocol, a replication package, and honest reporting of the trouble they had getting the tools to run. That genuinely hasn't been done before for this domain, and the paper earns its place on that count. The literature review part is mostly a replication of Bakhtin et al. with gray literature added, so that's incremental but fine.\n\nThe problem is the headline numbers. Table 5 shows each tool evaluated against a different ground-truth set: Code2DFD gets 727 items from 17 apps, RAD gets 391 from 13, MicroDepGraph 355 from 10. The F1 scores are then compared directly. If RAD's 13 apps happen to be the harder ones, its 0.79 could actually beat Code2DFD's 0.86 on a common subset. The paper never reports that comparison. The stress-test note is right: the \"Code2DFD is best\" claim is not supported by a controlled comparison.\n\nThe combination result has the same issue plus one more. The four-tool combination that hits F1=0.91 was selected by looking at the same data it is evaluated on. That's an in-sample optimum. Without a held-out validation or at least a clear statement that this is exploratory, the 0.91 overstates what a practitioner should expect.\n\nThere's also partial circularity in the endpoint ground truth. The authors extended the dataset by searching for @RequestMapping, @GetMapping, etc., which is exactly the mechanism several of the evaluated tools use to detect endpoints. For components and connections the DFD ground truth is more independent, so the endpoint comparison (RQ4.3) is the shakiest of the three.\n\nTo the authors' credit, most of these limitations are acknowledged in Section 6, and the transparency about execution failures is a model for this kind of work. But the abstract and conclusions still assert the ranking and the 0.91 as if they were robust.\n\nMy recommendation: send it to peer review, but require a common-subset comparison (restrict all tools to the apps and characteristics they all handle) and either a validation of the combination on held-out data or a rewrite that frames the combination as exploratory. With those changes, the paper would be a solid reference for practitioners choosing tools and for researchers building the next benchmark.","headline":"First execution-based comparison of microservice architecture recovery tools, but the headline rankings need a common-subset analysis before they can be believed.","tokens_in":31217,"tokens_out":3723,"would_cite":true,"duration_ms":38937,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper provides the first execution-based comparison of static architecture recovery tools, with Code2DFD best alone at F1 0.86 and a four-tool combination at 0.91.","keywords":["microservices","architecture recovery","static analysis","tool comparison","dataflow diagrams","precision recall F1","endpoint detection","registered report"],"falsifier":"Run one of the 17 benchmark applications and record the actual HTTP routes exposed by each service, for example by starting the service and enumerating registered mappings or reading runtime traffic, then compare that route list to the ground-truth DFD annotations. If any real endpoint is missing from the ground truth, or any ground-truth endpoint is not actually reachable, the reported precision, recall, and F1 values for endpoint extraction are wrong.","tokens_in":1504,"feed_emoji":"🧩","tokens_out":1640,"duration_ms":70024,"temperature":0.7,"pith_summary":"This paper tries to establish the first execution-based comparison of static architecture recovery tools for microservice applications, and to show both the accuracy of individual tools and whether combining tools improves recovery. The authors identify 13 tools through a multivocal literature review, succeed in running nine of them on 17 open-source Spring-based microservice applications, and measure precision, recall, and F1 against manually built dataflow-diagram ground truth. They report that Code2DFD is the best single tool with overall F1 = 0.86, and that merging the outputs of Code2DFD, MicroDepGraph, RAD, and RAD-source raises F1 to 0.91. If the finding holds, practitioners get a concrete reference for choosing tools, and researchers get evidence that no single tool yet recovers architecture perfectly.","feed_headline":"Combining four tools recovers microservice architecture at 0.91 F1","feed_subtitle":"First execution-based comparison finds Code2DFD best alone at 0.86; tool unions beat or match every single-tool result.","key_machinery":"The central mechanism is the evaluation pipeline: every tool is executed on the same 17 applications from the microSecEnD dataset, whose dataflow diagrams provide ground-truth components, connections, and endpoints; tool outputs are then manually compared to that ground truth to count true positives, false positives, and false negatives, from which precision, recall, and F1 are computed. For combinations, the paper merges per-characteristic results with logical AND and OR, so a characteristic counts as recovered under OR if any tool found it and under AND only if all tools found it. This pipeline is what makes the study an execution-based comparison rather than a review of advertised features.","core_discovery":"On the paper's own terms, the discovery is that static architecture recovery for microservices is now measurable on a common benchmark, and that the current best individual tool operates at overall F1 = 0.86. For components, Code2DFD reaches F1 = 0.98; for connections, Code2DFD scores 0.87; and for endpoints, RAD scores 0.79. Three of the nine runnable tools produced no useful output at all, and several others extracted almost nothing, while the tools that did work showed high precision but varying recall. Combining tool outputs with logical AND and OR on individual characteristics improves or matches the best individual result for almost every evaluated metric, with the four-tool combination of Code2DFD, MicroDepGraph, RAD, and RAD-source achieving overall F1 = 0.91.","pith_inferences":["The reported endpoint accuracy is likely an upper bound for non-Spring, non-Java stacks, because the ground truth models endpoints only through four Spring and Java annotations; a runtime-traffic-based endpoint list would probably reveal additional endpoints and shift the F1 scores.","The combination results are computed by merging outputs offline, not by an integrated pipeline; in practice a merging or voting layer would add integration overhead and could change the realized gains.","The paper's observation that no undetected characteristic is inherently undetectable suggests a single tool combining deployment-file parsing with deeper source or bytecode analysis could plausibly exceed the current 0.91 ceiling, a claim the authors hold qualitatively rather than demonstrate.","Extending the benchmark to non-Spring frameworks or RPC-based services would decide how far the 0.86 and 0.91 numbers generalize."],"forward_implications":["Code2DFD is the tool to beat across all three characteristics at once, with per-characteristic F1 scores of 0.98 for components, 0.87 for connections, and 0.66 for endpoints.","A practitioner who wants reliable endpoint extraction should prefer RAD (F1 = 0.79) over RAD-source (0.67) and Code2DFD (0.66).","Combining the outputs of Code2DFD, MicroDepGraph, RAD, and RAD-source raises overall F1 from 0.86 to 0.91, showing that tool synergy is real despite individual weaknesses.","Simple parsing of deployment files yields high precision quickly, but high recall requires deeper source analysis such as detecting Java annotations in the code.","For each evaluated metric and each characteristic, at least one tool combination matches or beats the best individual tool."],"supporting_citations":[{"why":"Supplies the microSecEnD dataset of 17 dataflow diagrams used as ground truth for components, connections, and endpoints.","marker":"[29]"},{"why":"The registered report version of this study whose peer-reviewed methodology the execution follows.","marker":"[19]"},{"why":"The systematic mapping study whose tool list and search string this work replicates and extends.","marker":"[20]"},{"why":"Describes Code2DFD, the best-performing individual tool in the comparison.","marker":"[36]"},{"why":"Describes MicroDepGraph, a deployment-file-based tool included in the best F1 combination.","marker":"[37]"},{"why":"Describes RAD and RAD-source, the two endpoint-detecting tools in the best F1 combination.","marker":"[43]"},{"why":"Presents the Attack Graph Generator, which achieved the highest individual recall and is used in combination evaluations.","marker":"[35]"},{"why":"Describes microMiner, the Kubernetes-parsing tool with high component precision in the comparison.","marker":"[38]"}],"fun_headline_variants":["First execution-based comparison of microservice recovery tools","Fusing four static tools recovers microservice architecture at 0.91 F1","Code2DFD leads individual tools; four-tool blend achieves 0.91 F1","Microservice recovery: 13 tools found, 9 tested, best blend at 0.91","Tool combination outperforms any single microservice recovery tool"],"cache_read_input_tokens":33408,"weakest_assumption_plain":"The comparison assumes the dataflow diagrams in the microSecEnD dataset are a complete and correct ground truth for components, connections, and endpoints, and that every real endpoint is declared via the Java annotations @RequestMapping, @PutMapping, @GetMapping, or @RepositoryRestResource; if the diagrams omit anything or endpoints exist through other mechanisms, every reported score shifts.","fun_headline_variants_meta":{"raw":{"variants":["First execution-based comparison of microservice recovery tools","Fusing four static tools recovers microservice architecture at 0.91 F1","Code2DFD leads individual tools; four-tool blend achieves 0.91 F1","Microservice recovery: 13 tools found, 9 tested, best blend at 0.91","Tool combination outperforms any single microservice recovery tool"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000698,"raw_usage":{"total_tokens":3153,"prompt_tokens":943,"completion_tokens":2210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":2109}},"tokens_in":559,"tokens_out":2210,"duration_ms":17165,"temperature":1.0,"reasoning_tokens":2109,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:53:28.402647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one of the 17 benchmark applications and record the actual HTTP routes exposed by each service, for example by starting the service and enumerating registered mappings or reading runtime traffic, then compare that route list to the ground-truth DFD annotations. If any real endpoint is missing from the ground truth, or any ground-truth endpoint is not actually reachable, the reported precision, recall, and F1 values for endpoint extraction are wrong.","supporting_citations":[{"cited_title":"microsecend: A dataset of security- enriched dataflow diagrams for microservice applications,","cited_arxiv_id":null,"evidence_quote":"Supplies the microSecEnD dataset of 17 dataflow diagrams used as ground truth for components, connections, and endpoints."},{"cited_title":"Tools reconstructing mi- croservice architecture: A systematic mapping study,","cited_arxiv_id":null,"evidence_quote":"The systematic mapping study whose tool list and search string this work replicates and extends."},{"cited_title":"Automatic extraction of security-rich dataflow diagrams for microservice applications written in java,","cited_arxiv_id":null,"evidence_quote":"Describes Code2DFD, the best-performing individual tool in the comparison."},{"cited_title":"A curated dataset of microservices-based systems,","cited_arxiv_id":null,"evidence_quote":"Describes MicroDepGraph, a deployment-file-based tool included in the best F1 combination."},{"cited_title":"On automated rbac assessment by constructing a centralized perspective for microservice mesh,","cited_arxiv_id":null,"evidence_quote":"Describes RAD and RAD-source, the two endpoint-detecting tools in the best F1 combination."},{"cited_title":"Attack graph generation for microservice architecture,","cited_arxiv_id":null,"evidence_quote":"Presents the Attack Graph Generator, which achieved the highest individual recall and is used in combination evaluations."},{"cited_title":"Mining the architecture of microservice-based applications from their kubernetes deployment,","cited_arxiv_id":null,"evidence_quote":"Describes microMiner, the Kubernetes-parsing tool with high component precision in the comparison."}],"review_version":1}