{"id":"761ee0ee-8483-4b1a-9417-ba6b26b8362d","arxiv_id":"2607.28331","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"After normalizing class-to-service mapping coverage, OpenAI o3 zero-shot and few-shot decompositions show identical structural adherence (TPD 68% PetClinic, 83.3% Bookstore).","lead":"LLM microservice proposals from text keep only part of a monolith’s real code dependencies; after fixing a mapping-coverage bias, zero-shot and few-shot look the same. The paper’s value is a static-analysis audit pipeline and a warning that unequal class mappings can fake strategy differences.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged mapping/coverage assumption.","rationale":"The strongest claim is narrow and well-supported: unequal mapping coverage can fabricate prompt-strategy differences; after intersection normalization, o3 zero-shot and few-shot tie on TPD/TVD for PetClinic and Bookstore. Tables 1–2 and the JWT/PetClinicApplication examples make the mechanism checkable. The reader's weakest assumption (intersection + partly manual mapping + static TVD/TPD as effort proxy) is exactly the condition the claim rests on; I do not find a more load-bearing internal gap. Small-N, single-LLM, and static-analysis limits justify keeping CONDITIONAL rather than upgrading to ACCEPT, but they do not warrant downgrading. Reproducibility via Zenodo further anchors the result. Hence agreement with the reader and no verdict change.","tokens_in":10034,"tokens_out":530,"duration_ms":8856,"concrete_test":"From the Zenodo package, re-run the pipeline on the published intersection CSVs for both systems; confirm analyzed edges = 25 (PetClinic) and 203 (Bookstore) and violation counts = 8 and 34 for both strategies. Then force-map the 13 omitted Bookstore JWT classes into zero-shot (e.g., as an auth service or nearest existing service) and recompute TVD/TPD; if equality breaks by more than a few points only under arbitrary forced mappings—and holds under the paper's intersection rule—the coverage-bias claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption already captures the load-bearing condition for the central claim: that intersection-normalized TVD/TPD on the static dependency graph (after heuristic+manual class-to-service mapping and limited framework-edge audit) is an unbiased enough universe to conclude zero-shot and few-shot are structurally equivalent. The paper is transparent about this—raw vs. normalized tables, the Bookstore JWT omission of 13 classes / 18 edges, and the PetClinicApplication singleton—and the equality result is a direct arithmetic consequence of recomputing over the same edge set (Table 2: identical violation counts). No hidden inconsistency or stronger internal flaw is evident; remaining limits (N=2 systems, one LLM, static-only edges, mild author-line circularity on the audited decompositions) are external-validity and construct-scope issues the reader already priced into CONDITIONAL, not defeaters of the narrow methodological punchline.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"This paper proposes an automated pipeline to structurally validate LLM-generated microservice decompositions against source-code dependencies extracted by static analysis. Using OpenAI o3 decompositions of PetClinic and Bookstore under zero-shot and few-shot prompting, it defines TVD/TPD metrics over a dependency graph and shows that raw differences between strategies disappear after restricting evaluation to the intersection of class-to-service mappings. Normalized TPD is 68.0% (PetClinic) and 83.3% (Bookstore) for both strategies. The central claim is methodological: structural comparisons of LLM decompositions must control mapping coverage, or apparent prompting effects may be artifacts of unequal class sets rather than architectural quality.","tokens_in":10215,"tokens_out":905,"duration_ms":13337,"significance":"If the result holds, the paper makes a useful and timely methodological contribution to LLM-assisted software architecture. It cleanly separates semantic plausibility of service boundaries from structural feasibility relative to an existing monolith, and it documents a concrete bias (unequal mapping coverage) that can invalidate prompt-strategy comparisons. Strengths include a transparent before/after robustness analysis (Tables 1–2), an auditable extract–map–violate–score pipeline with a public replication package, and appropriately cautious interpretation of TVD as migration-adaptation effort rather than absolute architectural quality. The scope is narrow (two Java systems, one model, static edges only), but the bias diagnosis is transferable and should influence how future empirical studies report structural metrics for AI-generated decompositions.","major_comments":[{"comment":"§3 Stage 3 and §4.1–4.2: the equality claim rests on intersection-normalized mappings after heuristic name/package matching plus manual refinement, including framework-mediated edges (e.g., Vet→Specialty). The paper should quantify residual mapping ambiguity (how many classes required manual adjudication; inter-rater checks if any) and state more explicitly that equivalence is established only on the commonly mapped subgraph. Without that, readers cannot judge how sensitive Table 2’s identical violation counts are to mapping choices.","section":"§3 Stage 3; §4.1 Table 2"},{"comment":"§3 Stage 2 and §7: the evaluated decompositions are taken from prior overlapping-author studies rather than regenerated under a fixed protocol in this work. That is acceptable for a validation-focused paper, but the manuscript should more clearly bound the claim—prompting strategies are structurally equivalent for these published o3 artifacts after coverage control—not that zero-shot and few-shot are generally equivalent for o3 or other models. A short sensitivity note on whether regenerating decompositions could change coverage patterns would strengthen internal validity.","section":"§3 Stage 2; §7 Internal validity"}],"minor_comments":[{"comment":"Figure 1 is referenced as the methodology overview but is not available in the text package; ensure the camera-ready figure clearly shows the four stages and where coverage normalization occurs.","section":"§3 Figure 1"},{"comment":"Formal definitions of TVD/TPD appear after the metrics are introduced narratively; moving the set-theoretic definition earlier in §3 Stage 4 would improve readability.","section":"§3 Stage 4"},{"comment":"Several bibliography entries are still anonymized placeholders ([1], [12]); replace with full citations for production.","section":"References"},{"comment":"Minor wording inconsistencies (e.g., “OpenAI o3” vs. model naming elsewhere; “PetClinicApplication” coverage detail) should be normalized for clarity.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Fit is appropriate for an empirical SE / software architecture venue. The contribution is genuinely methodological rather than a large-scale performance bake-off; I would not demand more systems as a condition of acceptance if the mapping-sensitivity reporting is tightened. Mild author-line overlap with the source decompositions is disclosed via citations and is not disqualifying for a validation pipeline paper, but editors may want consistency in de-anonymization of [1]/[12]."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is the bias finding, not a new decomposer. They take existing o3 zero/few-shot decompositions for PetClinic and Bookstore, build a Tree-sitter dependency graph, map classes to services, and score TVD/TPD. Raw numbers look different by prompt; after restricting to the intersection of mapped classes, the scores are identical (PetClinic TPD 68%, Bookstore 83.3%). The Bookstore JWT omission (13 classes, ~18 edges) is a concrete, checkable explanation. That is real evaluation hygiene for anyone comparing LLM architecture outputs.\n\nWhat is new is the attachable audit pipeline plus the explicit demonstration that unequal mapping coverage can manufacture a “winner.” Generation-from-requirements and code-driven decomposers already exist; this paper sits downstream and asks whether those proposals respect implemented edges. The before/after tables make the central claim easy to verify. Shipping a Zenodo package with graphs, mappings, and metrics is a genuine plus in this area.\n\nSoft spots are mostly scope, not internal contradiction. N=2 Java systems and one model; static edges only; class-to-service mapping still needs heuristics plus manual cleanup (they flag framework-mediated cases like Vet→Specialty). Intersection normalization removes coverage bias but also shrinks the evaluated universe, so “equivalent structural adherence” is conditional on that shared set. TVD/TPD measure migration friction against the monolith graph, not overall architectural quality—they say so. Mild circularity: they audit decompositions from an overlapping author line, but the metrics are grounded in an independent static graph, so that is not load-bearing fraud.\n\nMath is trivial proportions; data story is transparent; citations cover Newman/Fowler, reviews, and the generation/code-driven lines without obvious padding. This is for people who evaluate or adopt LLM-assisted monolith splits and want pre-migration structural checks, not for someone hunting a new clustering algorithm.\n\nI would send it to peer review. Expect referees to push external validity and mapping subjectivity, but the narrow claim is solid enough to deserve that time. Engage if you care about evaluation protocol for AI-generated architectures; skip if you only want generation methods.","headline":"Narrow but clean methodological result: coverage mismatch can fake prompt-strategy winners, and after intersection normalization o3 zero-shot and few-shot tie on structural adherence.","tokens_in":10875,"tokens_out":546,"would_cite":true,"duration_ms":12054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Once class-to-service mapping coverage is equalized, zero-shot and few-shot LLM microservice decompositions show identical structural adherence to source-code dependencies.","keywords":["Large Language Models","Microservice Architecture","Monolith Decomposition","Static Analysis","Structural Validation","Reproducibility","Prompting Strategies"],"falsifier":"Re-run the same pipeline on these systems (or larger ones) with complete, independently audited class-to-service mappings that cover every domain class for both strategies; a clear TPD advantage for one prompt style under full equal coverage would falsify the equivalence claim.","tokens_in":10882,"feed_emoji":"🧩","tokens_out":856,"duration_ms":29953,"temperature":0.7,"pith_summary":"Large language models can propose microservice splits from textual requirements that look coherent on paper, yet it is unclear whether those splits respect the real dependency structure already in the code. This paper builds an automated pipeline that extracts a static dependency graph from two Java monoliths, maps classes onto the proposed services, and measures how many dependencies stay inside a service versus cross a boundary. After forcing both prompting strategies onto exactly the same set of mapped classes, the apparent score gaps vanish: both preserve 68 percent of dependencies on PetClinic and 83 percent on Bookstore. The result matters because raw comparisons of prompting styles can be pure measurement artifacts; any structural evaluation of AI-generated architectures must control mapping coverage before claiming one approach is better.","feed_headline":"Mapping bias, not prompts, drove LLM split score gaps","feed_subtitle":"Equalize class coverage and zero-shot matches few-shot on real source dependencies.","key_machinery":"An automated structural validation pipeline that extracts a real dependency graph via static analysis, maps monolith classes to LLM-proposed services, flags cross-boundary edges as violations, and reports Total Violated Dependencies (TVD) and Total Preserved Dependencies (TPD = 1 − TVD), recomputed on the intersection of mapped classes to remove coverage bias.","core_discovery":"After normalizing to the intersection of class-to-service mappings, OpenAI o3 decompositions produced under zero-shot and few-shot prompting exhibit identical structural adherence on both systems—TPD 68.0 percent (TVD 32.0 percent) for PetClinic and TPD 83.3 percent (TVD 16.7 percent) for Bookstore. Differences seen in the raw metrics are entirely explained by unequal mapping coverage, not by genuine differences in architectural quality between the prompting strategies.","pith_inferences":["Coverage bias may systematically hide subsystems rich in dependencies (e.g., authentication), so mapping-coverage artifacts should become required reporting in this literature.","The same partial-assignment bias can distort other LLM software-engineering evaluations that map code entities to labels incompletely.","Feeding detected violations back as constraints in an iterative generation loop is a direct, testable way to close the remaining 17–32 percent preservation gap.","Pairing the static graph with light dynamic traces would better capture reflection and framework-mediated edges that static analysis misses."],"forward_implications":["Structural adherence metrics (TVD/TPD) should accompany semantic scores in benchmarks of LLM-based architecture generation.","Comparisons of prompting strategies must report class-to-service mapping coverage and normalize whenever coverages differ.","The pipeline can serve as a pre-migration audit that surfaces coupling hotspots and estimates structural adaptation effort.","Remaining violations are attributed more to missing structural input than to prompt style, favoring hybrid semantic-plus-structural methods over further prompt engineering alone."],"fun_headline_variants":["Mapping coverage, not prompts, erased LLM split gaps","Normalize class maps and zero-shot ties few-shot TPD","Raw prompt gaps vanish once mapping coverage is equalized","Coverage bias, not prompting, drove structural score gaps","Equal class coverage makes zero- and few-shot TPD identical"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That restricting the comparison to classes both strategies successfully mapped, after name heuristics and manual fixes for a few framework edges, yields a fair enough dependency set to conclude the prompting styles are structurally equivalent.","fun_headline_variants_meta":{"raw":{"variants":["Mapping coverage, not prompts, erased LLM split gaps","Normalize class maps and zero-shot ties few-shot TPD","Raw prompt gaps vanish once mapping coverage is equalized","Coverage bias, not prompting, drove structural score gaps","Equal class coverage makes zero- and few-shot TPD identical"]},"model":"grok-4.5","effort":"low","cost_usd":0.003526,"raw_usage":{"total_tokens":1122,"prompt_tokens":747,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":35264000,"prompt_tokens_details":{"text_tokens":747,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":310,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":747,"tokens_out":65,"duration_ms":4880,"temperature":1.0,"reasoning_tokens":310,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T11:04:36.307663+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same pipeline on these systems (or larger ones) with complete, independently audited class-to-service mappings that cover every domain class for both strategies; a clear TPD advantage for one prompt style under full equal coverage would falsify the equivalence claim.","supporting_citations":[],"review_version":1}