{"id":"7c5cd8bd-0b84-47dc-8fe4-2401cd5194cd","arxiv_id":"2509.03463","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An empirical study of an LLM critique-refine pipeline for activity diagrams finds algorithmic structural checks beat LLM checks, but measurement design issues cloud the results.","lead":"LADEX is a pipeline that generates activity diagrams from natural-language text using an LLM draft plus a critique-and-fix loop, tested on 220 process descriptions. The paper reports that fixed algorithmic structural checks are more reliable than LLM structural checks, while adding LLM semantic checks rarely changes the outcome.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic evaluation is underspecified: B-Match has no similarity threshold and unsound outputs are not handled, so the Alg-vs-LLM quality claim is not reproducible as stated.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the semantic evaluation is not a controlled, well-defined comparison. The paper's headline numbers—86% correctness and 92% completeness for the best variant—are computed by matchers that are specified only for structurally sound inputs, while the experimental protocol accepts outputs that may be unsound for all non-algorithmic variants. The paper never says how unsound outputs are treated in the semantic metrics, and the B-Match algorithm as written matches any successor with non-negative similarity, making the reported scores impossible to reproduce without an additional undocumented filter. This is not merely a stylistic omission: RQ2 and RQ3's correctness/completeness improvements could be artifacts of filtering or of threshold-free matching. The structural-consistency result is also partly tautological for the algorithmic variants, since accepted outputs are required to pass the algorithmic critique, although the nonzero violation rates for LLM-based critique remain meaningful. A single re-analysis with an explicit threshold and a uniform unsound-output policy would settle whether the claimed advantage is real. Because this concern directly supports the reader's REJECT verdict, no change to the verdict is needed.","tokens_in":32699,"tokens_out":8961,"duration_ms":96524,"concrete_test":"Re-run Tables VIII and IX with the released implementation after (i) adding a documented B-Match similarity threshold (e.g., cosine thresholds 0.5 and 0.7) and (ii) applying one explicit, uniform policy for unsound outputs—either scoring all outputs for every variant or scoring only the structurally sound subset for every variant. The RQ3 quantitative claim survives only if the algorithmic variants still significantly outperform the LLM-critique variants under both corrected protocols.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central RQ3 conclusion—that algorithmic structural checking yields more structurally consistent diagrams and higher semantic correctness/completeness than LLM-based structural checking—depends on the semantic scores being well-defined and comparable across variants. Section IV-A and IV-B state that B-Match and L-Match take two structurally sound activity diagrams as input, yet Table VII shows that Baseline, LADEX-LLM-LLM, and LADEX-LLM-NA produce many unsound outputs, and Section V-D never states whether those unsound outputs are excluded from Tables VIII/IX. If they are included, the matchers are applied outside their defined input domain; if they are excluded, the Alg variants are scored only on outputs guaranteed to be sound by the accept-only-if-pass protocol, while the LLM-critique variants are scored on a mix of sound and unsound outputs, confounding the comparison. Independently, Algorithm 1 has no similarity threshold: bestScore is initialized to 0.0 and any successor with non-negative simStep is enqueued, so correctness/completeness can be inflated by low-similarity matches. Without an undocumented filter, the reported 40-90% correctness values in Table VIII cannot be reproduced from the described algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LADEX, an LLM-based pipeline that generates activity diagrams from natural-language process descriptions through an iterative critique-and-refine loop. Five variants are compared: a Baseline with no loop, and four loop variants that combine either algorithmic or LLM-based structural checking with or without LLM-based alignment checking. Evaluation is performed on a 20-document industrial dataset from Ciena and a 200-document subset of the PAGED benchmark, using two node-matching methods (a trace-based behavioural matcher, B-Match, and an LLM-based matcher, L-Match) to derive semantic correctness and completeness scores, along with structural-consistency and cost metrics. The central claims are that the critique-refine loop improves quality over single-pass generation, that algorithmic structural checking yields structurally consistent diagrams and higher correctness/completeness than LLM-based structural checking, and that the best configuration combines algorithmic structural checks with LLM-based alignment checks.","tokens_in":32892,"tokens_out":5075,"duration_ms":57505,"significance":"If the empirical claims held, the paper would be a useful systematic study of critique-refine strategies for LLM-based behavioural model generation, with an unusual industrial dataset and two complementary evaluation instruments. The ablation design (structural checks algorithmic vs. LLM; alignment checks on vs. off) is well motivated, and the use of multiple LLMs, two datasets, and statistical testing with effect sizes is a strength. However, the main RQ3 conclusion is currently not supported as stated: the algorithmic variants' zero structural violations follow from the accept-only-if-pass protocol rather than from empirical observation, and the B-Match algorithm as presented cannot reproduce the reported correctness scores without an undocumented similarity threshold. These issues affect the central comparative claims and require substantial re-analysis or re-framing.","major_comments":[{"comment":"The experimental protocol in Section V-D states: 'If the loop does not converge within these five iterations ... we discard the activity diagram and restart the variant from its generation step' and 'an activity diagram produced by a given LADEX variant is accepted as that variant's output only if it passes the critique check.' For LADEX-Alg-LLM and LADEX-Alg-NA, the structural critique is a deterministic algorithm, so the zero-violation entries in Table VII are guaranteed by construction, not measured. Baseline, by contrast, has no such acceptance filter and no discarded outputs. This asymmetric filtering confounds the RQ3 structural-consistency answer and also contaminates semantic scores: low-quality outputs are removed only from the algorithmic variants (and, to a lesser extent, from LLM-critique variants that happen to pass their LLM critique). The authors should report results unde","section":"Section V-D, Table VII; RQ2/RQ3"},{"comment":"As written, Algorithm 1 has no similarity threshold. Lines 12-17 initialize bestScore to 0.0 and accept any successor s' with score >= bestScore; line 15 uses >=, so a successor with simStep = 0.0 is selected. Since simStep is a cosine-based similarity that can be low or negative, every reachable successor of a matched source node is matched to some target successor. For a structurally sound source diagram (which is fully connected by SC6), all source nodes are reachable and hence all become matched, forcing correctness near 100%. The reported B-Match correctness values in Table VIII(b), which range from about 32% to 90%, are therefore impossible to reproduce from the described algorithm without an undocumented filtering rule. The authors must specify the threshold, state how negative and zero similarities are handled, and justify the threshold's effect on the reported scores.","section":"Section IV-A, Algorithm 1; Tables VIII(b), IX(b)"},{"comment":"Both matchers are defined for structurally sound activity diagrams: Section IV-A says 'B-Match takes as input two structurally sound activity diagrams' and Section IV-B says 'L-Match ... takes two structurally sound activity diagrams as input.' Yet Table VII shows that Baseline, LADEX-LLM-LLM, and LADEX-LLM-NA produce many structurally unsound outputs, and the paper never states whether those unsound diagrams are excluded from the semantic correctness/completeness computations in Tables VIII and IX. If they are included, the matchers are applied outside their specified input domain. If they are excluded, the algorithmic variants are scored only on the subset that passed the algorithmic critique, while the LLM-based variants are scored on a mixture of sound and unsound outputs (or on a differently filtered subset). Either way, the Alg-versus-LLM semantic comparison is confounded. The auth","section":"Sections IV-A, IV-B, V-B, V-E"}],"minor_comments":[{"comment":"Typo: 'acticity diagram' should be 'activity diagram'.","section":"Section V-D"},{"comment":"The caption says 'For the Industry dataset, 100 activity diagrams were generated in total' and 'For the PAGED dataset, 1000 activity diagrams were generated in total,' but the table has one row per variant-LLM pair. The intended denominator per cell should be stated clearly; as written, the 'in total' phrasing is ambiguous and inconsistent with the 16,000 submitted process descriptions mentioned in Section V-D.","section":"Table VII caption and Section V-A"},{"comment":"The L-Match validation is based on only five activity diagrams per dataset. Please report the selection procedure, per-diagram precision/recall, and the experience/blinding of the annotators. This is relevant because L-Match is one of the two primary evaluation instruments for all semantic scores.","section":"Section IV-B, Table V"},{"comment":"Typo in caption: 'L-L-Match' should be 'L-Match'.","section":"Figure 9 caption"},{"comment":"The random subset of PAGED is described as 'randomly selected 200 entries,' but no seed or selection details are given. Reporting the seed or the sampling procedure would aid reproducibility.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practically relevant question, and the dataset scope is a strength. However, the central comparison in RQ3 is currently undermined by the acceptance-by-construction protocol and by the underspecified B-Match algorithm. These are fixable in principle with additional experiments or careful re-analysis, but the present manuscript does not support its headline claims as written. If the authors cannot provide the required re-analysis or re-framing, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the question is right and the ablation design is sensible, but the central quantitative claims are built on a filtering rule and an underspecified matcher. The RQ3 result that algorithmic structural checks achieve structural consistency is a tautology, and B-Match as written cannot produce the reported correctness values.\n\nWhat is genuinely useful: this is the first systematic ablation of algorithmic versus LLM structural critics inside a critique–refine loop for activity diagrams. The five-variant design is clean, the two datasets (one industrial, one public) add credibility, and the cost analysis in terms of LLM calls is a practical angle most model-generation papers ignore. The L-Match validation against expert annotations is a nice touch, and the UML-derived constraints are reasonable.\n\nThe soft spots are not minor. First, Section V-D implements an accept-only-if-pass rule: any variant with a refinement loop discards diagrams that do not converge within five iterations and restarts. Baseline has no such filter, so the RQ2 comparison conflates the effect of the loop with the effect of discarding failures. Second, the Alg variants accept only diagrams that pass the algorithmic structural check, so their zero-violation record in Table VII is constructional, not empirical. Third, B-Match (Algorithm 1) has no similarity threshold: bestScore starts at 0.0 and any non-negative simStep is accepted, so every node in a connected source diagram gets matched to some target node. Under that algorithm, correctness should be 100% for every structurally sound diagram, which contradicts the reported 40–90% values. There must be an undocumented threshold or post-filter, and without it the semantic scores are not reproducible. Fourth, B-Match and L-Match are defined for structurally sound inputs, but the paper scores unsound outputs from Baseline and LLM-critique variants without stating whether they are excluded. If included, the matchers operate outside their domain; if excluded, the Alg variants are compared only on the filtered subset, again confounding the comparison.\n\nWho this is for: researchers working on LLM-based model generation, especially those interested in critique–refine loops and evaluation methodology. The paper would be a good case study in how selection effects and matcher specifications can undermine empirical claims. With a revised protocol, a specified B-Match threshold, and a re-analysis of RQ2/RQ3 that separates filtering from refinement, the study could be publishable. As is, I would not trust the headline numbers.\n\nRecommendation: send it out for peer review — the question is timely and the flaws are fixable — but reviewers should demand the fixes before acceptance.","headline":"Useful ablation design, but the headline comparisons are undercut by an asymmetric discard-and-restart rule and an underspecified B-Match; the structural-consistency claim for algorithmic critics is constructional.","tokens_in":33459,"tokens_out":4102,"would_cite":false,"duration_ms":41219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that when LLMs generate activity diagrams from natural-language process descriptions, the critique-refine loop improves quality only if structural well-formedness checks are done algorithmically, not by the LLM itself.","keywords":["activity diagrams","LLM-based model generation","critique-refine loop","structural constraints","semantic alignment","behavioural matching","natural language to model","automated evaluation"],"falsifier":"Run the same five LADEX variants on a fresh set of process descriptions, swapping only the structural critic between algorithmic and LLM. If the LLM-critiqued outputs show no structural violations in repeated runs, or if B-Match's node scores on structurally unsound diagrams turn out to be arbitrary because no matching threshold is defined, the central comparison collapses.","tokens_in":1540,"feed_emoji":"⚙️","tokens_out":1977,"duration_ms":71224,"temperature":0.7,"pith_summary":"This paper argues that when LLMs generate activity diagrams from natural-language process descriptions, an iterative critique-refine loop improves quality, but only if the structural well-formedness checks are done by deterministic algorithms rather than by the LLM. The paper introduces LADEX, a pipeline that generates an initial diagram, critiques it against six structural and five alignment constraints, and refines it until clean. Across two datasets and three LLMs, the algorithmic-critique variants produce zero structurally invalid diagrams, while LLM-critique variants leave many diagrams invalid. The best configuration combines algorithmic structural checks with LLM-based semantic alignment checks and reaches about 86% correctness and 92% completeness at under five LLM calls per diagram. The practical claim is that rule-based validation should be delegated to code, leaving LLMs to judge meaning.","feed_headline":"Algorithmic critics beat LLM self-review in diagram refinement","feed_subtitle":"Rule-based structural checks produce sound activity diagrams; LLM checks often leave them broken.","key_machinery":"The critique-refine loop with two kinds of critics. Structural constraints SC1-SC6, derived from the UML 2.5.1 well-formedness rules, define what counts as a valid activity diagram; the algorithmic critic checks them deterministically, while the LLM critic performs the same checks by prompting. Alignment constraints AC1-AC5 encode how the diagram should reflect the process description. B-Match operationalizes trace-based behavioural similarity through a breadth-first, simulation-like node matching with label and step similarity, while L-Match is an LLM-based node matcher. The central mechanism is the separation of structural (rule-checkable) from semantic (judgment-requiring) critique inside","core_discovery":"The paper claims that, for activity diagram generation from natural language, the critique-refine loop is beneficial, and that the way structural checks are performed determines whether refinement converges to well-formed models. In the experiments, variants where structural constraints are checked algorithmically yielded 100% structurally consistent diagrams, whereas variants using an LLM as structural critic still produced inconsistent diagrams in a substantial fraction of cases. Algorithmic structural checking improved correctness by 16.95% and completeness by 15.12% over LLM structural checking. Adding LLM semantic alignment checks gave a significant correctness gain on one dataset only;","pith_inferences":["If this result generalizes beyond activity diagrams, any LLM pipeline generating models with formal syntax, such as statecharts, BPMN, or sequence diagrams, should outsource well-formedness checks to a deterministic validator rather than relying on LLM self-review.","The observed failure of LLM structural critics may be specific to prompt-only enforcement of rules; a testable extension is to give the LLM critic an executable checking tool and compare against the paper's numbers.","The cost pattern points toward a neuro-symbolic division of labour: the LLM proposes structures, a deterministic verifier disposes of invalid ones, and the LLM refines only what the verifier flags.","Because the evaluation uses a single ground-truth diagram per process description, real tasks with multiple legitimate modelling choices could penalize correct alternatives; multi-reference evaluation would be a stronger test of the central claim."],"forward_implications":["Critique-refine loops beat single-pass generation on structural validity, semantic correctness, and completeness for LLM-generated activity diagrams.","Structural well-formedness should be enforced by deterministic algorithms, not by asking an LLM to self-critique; LLM structural critics leave many diagrams invalid and lower correctness and completeness.","The best quality-cost trade-off combines algorithmic structural checks with LLM-based semantic checks using a reasoning LLM, reaching about 86% correctness and 92% completeness at under five LLM calls.","If minimizing LLM calls is the priority, algorithmic structural checks alone nearly match that quality at about one LLM call per diagram.","The two automated evaluation methods, trace-based behavioural matching and LLM-based matching, agree on variant comparisons, supporting automated quality assessment without manual expert scoring."],"supporting_citations":[{"why":"The UML 2.5.1 standard from which the paper derives its six structural constraints SC1-SC6.","marker":"[3]"},{"why":"Formalizes activity-diagram behaviour as traces, the basis of the B-Match similarity measure.","marker":"[12]"},{"why":"Earlier quantitative simulation-based matching method that B-Match adapts from state machines to activity diagrams.","marker":"[13]"},{"why":"Extends the same matching and merging approach to variant feature specifications, another source B-Match builds on.","marker":"[14]"},{"why":"Public benchmark of text-diagram pairs supplying the 200 examples used in the public-domain experiments.","marker":"[15]"},{"why":"Prior iterative LLM element extraction and refinement study, the main existing critique-refine-style baseline.","marker":"[10]"},{"why":"Sentence-embedding method used by B-Match's simLabel for semantic label similarity.","marker":"[25]"},{"why":"Instruction-following LLM used as one of the generators and critics under test.","marker":"[16]"},{"why":"Reasoning-based LLM used for generation, critique, and as the LLM matcher.","marker":"[17]"},{"why":"Open reasoning LLM used on the public dataset as a third generation and critique model.","marker":"[18]"}],"fun_headline_variants":["Algorithmic critics beat LLM self-review in diagram refinement","Rule-based checks produce sounder diagrams than LLM critics","LLM self-review leaves diagram flaws that algorithms catch","For activity diagrams, algorithmic critique beats LLM critique","Structural validity hinges on critique method, not just loop"],"cache_read_input_tokens":35200,"weakest_assumption_plain":"The results assume the automated matchers score all generated diagrams fairly, even ones that are structurally invalid; if a structurally invalid diagram receives an artificially low or high score, the reported correctness and completeness gaps could be distorted.","fun_headline_variants_meta":{"raw":{"variants":["Algorithmic critics beat LLM self-review in diagram refinement","Rule-based checks produce sounder diagrams than LLM critics","LLM self-review leaves diagram flaws that algorithms catch","For activity diagrams, algorithmic critique beats LLM critique","Structural validity hinges on critique method, not just loop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1252,"prompt_tokens":834,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":340}},"tokens_in":578,"tokens_out":418,"duration_ms":3895,"temperature":1.0,"reasoning_tokens":340,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:52:28.929231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five LADEX variants on a fresh set of process descriptions, swapping only the structural critic between algorithmic and LLM. If the LLM-critiqued outputs show no structural violations in repeated runs, or if B-Match's node scores on structurally unsound diagrams turn out to be arbitrary because no matching threshold is defined, the central comparison collapses.","supporting_citations":[{"cited_title":"Unified modeling language (UML) version 2.5.1,","cited_arxiv_id":null,"evidence_quote":"The UML 2.5.1 standard from which the paper derives its six structural constraints SC1-SC6."},{"cited_title":"Matching and merging of statecharts specifications,","cited_arxiv_id":null,"evidence_quote":"Earlier quantitative simulation-based matching method that B-Match adapts from state machines to activity diagrams."},{"cited_title":"Matching and merging of variant feature specifications,","cited_arxiv_id":null,"evidence_quote":"Extends the same matching and merging approach to variant feature specifications, another source B-Match builds on."},{"cited_title":"PAGED: A benchmark for procedural graphs extraction from documents,","cited_arxiv_id":null,"evidence_quote":"Public benchmark of text-diagram pairs supplying the 200 examples used in the public-domain experiments."}],"review_version":1}