Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

The Impact of Critique on LLM-Based Model Generation from Natural Language: The Case of Activity Diagrams

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that when LLMs generate activity diagrams from natural-language process descriptions, the critique-refine loop improves quality only if structural well-formedness checks are done algorithmically, not by the LLM itself.

desk verdict Useful ablation design, but the headline comparisons are undercut by an asymmetric discard-and-restart rule and an underspecified B-Match; the structural-consistency claim for algorithmic critics is constructional. read the letter →

arxiv 2509.03463 v3 pith:QLE6T2TJ submitted 2025-09-03 cs.SE

Parham Khamsepour , Mark Cole , Ish Ashraf , DaYuan Tan , Sandeep Puri , Mehrdad Sabetzadeh , Shiva Nejati This is my paper · ORCID
classification cs.SE
keywords activitydiagramsLLM-basedmodelgenerationcritique-refineloopstructuralconstraintssemanticalignmentbehaviouralmatchingnaturallanguagetoautomatedevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that when LLMs generate activity diagrams from natural-language process descriptions, an iterative critique-refine loop improves quality, but only if the structural well-formedness checks are done by deterministic algorithms rather than by the LLM. The paper introduces LADEX, a pipeline that generates an initial diagram, critiques it against six structural and five alignment constraints, and refines it until clean. Across two datasets and three LLMs, the algorithmic-critique variants produce zero structurally invalid diagrams, while LLM-critique variants leave many diagrams invalid. The best configuration combines algorithmic structural checks with LLM-based semantic alignment checks and reaches about 86% correctness and 92% completeness at under five LLM calls per diagram. The practical claim is that rule-based validation should be delegated to code, leaving LLMs to judge meaning.

What carries the argument

The critique-refine loop with two kinds of critics. Structural constraints SC1-SC6, derived from the UML 2.5.1 well-formedness rules, define what counts as a valid activity diagram; the algorithmic critic checks them deterministically, while the LLM critic performs the same checks by prompting. Alignment constraints AC1-AC5 encode how the diagram should reflect the process description. B-Match operationalizes trace-based behavioural similarity through a breadth-first, simulation-like node matching with label and step similarity, while L-Match is an LLM-based node matcher. The central mechanism is the separation of structural (rule-checkable) from semantic (judgment-requiring) critique inside

What would settle it

Run the same five LADEX variants on a fresh set of process descriptions, swapping only the structural critic between algorithmic and LLM. If the LLM-critiqued outputs show no structural violations in repeated runs, or if B-Match's node scores on structurally unsound diagrams turn out to be arbitrary because no matching threshold is defined, the central comparison collapses.

Watch

Extended reading notes

Core claim

The paper claims that, for activity diagram generation from natural language, the critique-refine loop is beneficial, and that the way structural checks are performed determines whether refinement converges to well-formed models. In the experiments, variants where structural constraints are checked algorithmically yielded 100% structurally consistent diagrams, whereas variants using an LLM as structural critic still produced inconsistent diagrams in a substantial fraction of cases. Algorithmic structural checking improved correctness by 16.95% and completeness by 15.12% over LLM structural checking. Adding LLM semantic alignment checks gave a significant correctness gain on one dataset only;

Load-bearing premise

The results assume the automated matchers score all generated diagrams fairly, even ones that are structurally invalid; if a structurally invalid diagram receives an artificially low or high score, the reported correctness and completeness gaps could be distorted.

Editorial extensions

If this is right

  • Critique-refine loops beat single-pass generation on structural validity, semantic correctness, and completeness for LLM-generated activity diagrams.
  • Structural well-formedness should be enforced by deterministic algorithms, not by asking an LLM to self-critique; LLM structural critics leave many diagrams invalid and lower correctness and completeness.
  • The best quality-cost trade-off combines algorithmic structural checks with LLM-based semantic checks using a reasoning LLM, reaching about 86% correctness and 92% completeness at under five LLM calls.
  • If minimizing LLM calls is the priority, algorithmic structural checks alone nearly match that quality at about one LLM call per diagram.
  • The two automated evaluation methods, trace-based behavioural matching and LLM-based matching, agree on variant comparisons, supporting automated quality assessment without manual expert scoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this result generalizes beyond activity diagrams, any LLM pipeline generating models with formal syntax, such as statecharts, BPMN, or sequence diagrams, should outsource well-formedness checks to a deterministic validator rather than relying on LLM self-review.
  • The observed failure of LLM structural critics may be specific to prompt-only enforcement of rules; a testable extension is to give the LLM critic an executable checking tool and compare against the paper's numbers.
  • The cost pattern points toward a neuro-symbolic division of labour: the LLM proposes structures, a deterministic verifier disposes of invalid ones, and the LLM refines only what the verifier flags.
  • Because the evaluation uses a single ground-truth diagram per process description, real tasks with multiple legitimate modelling choices could penalize correct alternatives; multi-reference evaluation would be a stronger test of the central claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents LADEX, an LLM-based pipeline that generates activity diagrams from natural-language process descriptions through an iterative critique-and-refine loop. Five variants are compared: a Baseline with no loop, and four loop variants that combine either algorithmic or LLM-based structural checking with or without LLM-based alignment checking. Evaluation is performed on a 20-document industrial dataset from Ciena and a 200-document subset of the PAGED benchmark, using two node-matching methods (a trace-based behavioural matcher, B-Match, and an LLM-based matcher, L-Match) to derive semantic correctness and completeness scores, along with structural-consistency and cost metrics. The central claims are that the critique-refine loop improves quality over single-pass generation, that algorithmic structural checking yields structurally consistent diagrams and higher correctness/completeness than LLM-based structural checking, and that the best configuration combines algorithmic structural checks with LLM-based alignment checks.

Significance. If the empirical claims held, the paper would be a useful systematic study of critique-refine strategies for LLM-based behavioural model generation, with an unusual industrial dataset and two complementary evaluation instruments. The ablation design (structural checks algorithmic vs. LLM; alignment checks on vs. off) is well motivated, and the use of multiple LLMs, two datasets, and statistical testing with effect sizes is a strength. However, the main RQ3 conclusion is currently not supported as stated: the algorithmic variants' zero structural violations follow from the accept-only-if-pass protocol rather than from empirical observation, and the B-Match algorithm as presented cannot reproduce the reported correctness scores without an undocumented similarity threshold. These issues affect the central comparative claims and require substantial re-analysis or re-framing.

major comments (3)
  1. [Section V-D, Table VII; RQ2/RQ3] The experimental protocol in Section V-D states: 'If the loop does not converge within these five iterations ... we discard the activity diagram and restart the variant from its generation step' and 'an activity diagram produced by a given LADEX variant is accepted as that variant's output only if it passes the critique check.' For LADEX-Alg-LLM and LADEX-Alg-NA, the structural critique is a deterministic algorithm, so the zero-violation entries in Table VII are guaranteed by construction, not measured. Baseline, by contrast, has no such acceptance filter and no discarded outputs. This asymmetric filtering confounds the RQ3 structural-consistency answer and also contaminates semantic scores: low-quality outputs are removed only from the algorithmic variants (and, to a lesser extent, from LLM-critique variants that happen to pass their LLM critique). The authors should report results unde
  2. [Section IV-A, Algorithm 1; Tables VIII(b), IX(b)] As written, Algorithm 1 has no similarity threshold. Lines 12-17 initialize bestScore to 0.0 and accept any successor s' with score >= bestScore; line 15 uses >=, so a successor with simStep = 0.0 is selected. Since simStep is a cosine-based similarity that can be low or negative, every reachable successor of a matched source node is matched to some target successor. For a structurally sound source diagram (which is fully connected by SC6), all source nodes are reachable and hence all become matched, forcing correctness near 100%. The reported B-Match correctness values in Table VIII(b), which range from about 32% to 90%, are therefore impossible to reproduce from the described algorithm without an undocumented filtering rule. The authors must specify the threshold, state how negative and zero similarities are handled, and justify the threshold's effect on the reported scores.
  3. [Sections IV-A, IV-B, V-B, V-E] Both matchers are defined for structurally sound activity diagrams: Section IV-A says 'B-Match takes as input two structurally sound activity diagrams' and Section IV-B says 'L-Match ... takes two structurally sound activity diagrams as input.' Yet Table VII shows that Baseline, LADEX-LLM-LLM, and LADEX-LLM-NA produce many structurally unsound outputs, and the paper never states whether those unsound diagrams are excluded from the semantic correctness/completeness computations in Tables VIII and IX. If they are included, the matchers are applied outside their specified input domain. If they are excluded, the algorithmic variants are scored only on the subset that passed the algorithmic critique, while the LLM-based variants are scored on a mixture of sound and unsound outputs (or on a differently filtered subset). Either way, the Alg-versus-LLM semantic comparison is confounded. The auth
minor comments (5)
  1. [Section V-D] Typo: 'acticity diagram' should be 'activity diagram'.
  2. [Table VII caption and Section V-A] The caption says 'For the Industry dataset, 100 activity diagrams were generated in total' and 'For the PAGED dataset, 1000 activity diagrams were generated in total,' but the table has one row per variant-LLM pair. The intended denominator per cell should be stated clearly; as written, the 'in total' phrasing is ambiguous and inconsistent with the 16,000 submitted process descriptions mentioned in Section V-D.
  3. [Section IV-B, Table V] The L-Match validation is based on only five activity diagrams per dataset. Please report the selection procedure, per-diagram precision/recall, and the experience/blinding of the annotators. This is relevant because L-Match is one of the two primary evaluation instruments for all semantic scores.
  4. [Figure 9 caption] Typo in caption: 'L-L-Match' should be 'L-Match'.
  5. [Section V-C] The random subset of PAGED is described as 'randomly selected 200 entries,' but no seed or selection details are given. Reporting the seed or the sampling procedure would aid reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Structural-consistency result for algorithmic variants is guaranteed by the accept-only-if-pass protocol, making part of RQ3 definitional; semantic comparisons remain empirical.

  1. self definitional [Section V-D (Experimental Procedure) and Section V-E (RQ3 result)]
    "If the loop does not converge within these five iterations, i.e., if the critique step continues to identify issues, we discard the acticity diagram and restart the variant from its generation step. In other words, an activity diagram produced by a given LADEX variant is accepted as that variant’s output only if it passes the critique check."

    The paper's RQ3 answer claims as a finding: 'activity diagrams refined based on algorithmic structural checks achieve structural consistency, whereas those refined based on LLM-based checks often still show structural inconsistencies.' For LADEX-Alg-NA and LADEX-Alg-LLM, structural checks in the critique are algorithmic (Table III), so passing the critique means passing the algorithmic structural checker. The acceptance rule in V-D therefore definitionally guarantees every Alg output is structurally sound by that checker. Table VII's zero-violation rows for Alg variants and the RQ3 structural-consistency claim are the acceptance filter restated, not an empirical observation. LLM-critique variants, by contrast, can be accepted with residual structural violations because their LLM critic may

full rationale

The paper's main content is an empirical evaluation, not an analytic derivation, and most of it is externally grounded: semantic correctness/completeness is measured against expert ground truths, L-Match is validated against expert matchings, and B-Match is specified in full. There is no fitted-parameter circularity, no load-bearing self-citation chain, and no ansatz smuggled in via citation. The one definitional circularity is the structural-consistency claim for algorithmic variants: Section V-D defines an accepted output as one that passed the critique, and for Alg variants the critique includes the algorithmic structural checks. Hence Table VII's zero-violation rows for LADEX-Alg-* and the RQ3 statement that algorithmic variants 'achieve structural consistency' restate the acceptance filter rather than measuring an empirical effect. The semantic improvements of Alg over LLM structural checking are partly empirical, but they are confounded by the same filter: Alg outputs are guaranteed structurally sound while LLM-critic outputs may include unsound diagrams, even though B-Match and L-Match are defined only for structurally sound inputs. These latter points are validity threats rather than circularities, and they are noted here only to delimit the scope of the circularity finding. Overall, the central RQ3 structural-consistency claim reduces by construction, while the semantic comparison retains independent empirical content, so a score of 6 is appropriate.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The pipeline rests on domain assumptions about what structural correctness means, how behavior should be compared, and the validity of a single ground truth per document. No new physical or formal entities are postulated. B-Match and L-Match are measurement procedures rather than invented entities. The matching and cost numbers depend on implementation choices that are not fully specified.

free parameters (4)
  • refinement iteration cap = 5
    Section V-D caps refinement at five iterations; non-converging diagrams are discarded and restarted, so this design choice directly affects results.
  • GPT-4.1 Mini temperature = 0.0
    Section V-F sets temperature to 0.0 for the instruction-following LLM to reduce randomness; this affects output variability and comparisons across LLMs.
  • DeepSeek temperature = 0.6
    Section V-F uses the recommended temperature 0.6 for DeepSeek-R1-Distill-Llama-70B; a chosen setting that influences generated outputs.
  • B-Match similarity threshold
    Algorithm 1 does not state a threshold or stopping condition. Without one, every node is matched to a best-scoring successor, so the reported correctness values imply an undisclosed threshold.
assumptions (5)
  • domain assumption An activity diagram is a tuple <NL, TL, N, T> with nodes partitioned into action, decision, initial, and end nodes, with fork and join abstracted as action nodes (Definition 2.1).
    This formalization underlies all generated outputs and both matching methods; it is a modeling choice, not a standard UML execution semantics.
  • domain assumption Structural consistency is fully captured by constraints SC1-SC6 derived from the UML 2.5.1 well-formedness rules.
    Section II reduces 17 UML constraints to six enforceable ones and treats them as the definition of structural correctness. This is a judgment call that shapes all structural results.
  • standard math Trace-based operational semantics characterize activity diagram behavior, and simulation-style matching captures behavioral similarity.
    Section IV-A builds on prior trace-based semantics and simulation preorders (Maoz et al., Sokolsky et al.) rather than proving them.
  • domain assumption Each process description has a single valid ground-truth activity diagram.
    Section V-F acknowledges this assumption; in practice a description may admit multiple valid models depending on abstraction and grouping choices.
  • domain assumption LLM outputs in the specified Draw.io CSV format can be parsed reliably into the formal activity-diagram model.
    The pipeline relies on the LLM producing valid CSV; parsing failures or silent malformations are not reported separately and would affect all metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impact of Critique on LLM-Based Model Generation from Natural Language: The Case of Activity Diagrams." pith.science (2026). https://pith.science/paper/QLE6T2TJ

@misc{pith2026250903463,
  author       = {Pith},
  title        = {Pith review of: The Impact of Critique on LLM-Based Model Generation from Natural Language: The Case of Activity Diagrams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QLE6T2TJ}},
  note         = {Machine review of arXiv:2509.03463}
}
read the original abstract

Large Language Models (LLMs) show strong potential for automating model generation from natural-language descriptions. A common approach begins with an initial model generation, followed by an iterative critique-refine loop in which the model is evaluated for issues and refined based on those issues. This process needs to address: (1) structural correctness -- compliance with well-formedness rules -- and (2) semantic alignment -- accurate reflection of the intended meaning in the source text. We present LADEX (LLM-based Activity Diagram Extractor), a pipeline for deriving activity diagrams from natural-language process descriptions using an LLM-driven critique-refine process. Structural checks in LADEX can be performed either algorithmically or by an LLM, while alignment checks are performed by an LLM. We design five ablated variants of LADEX to study: (i) the impact of the critique-refine loop itself, (ii) the role of LLM-based semantic checks, and (iii) the comparative effectiveness of algorithmic versus LLM-based structural checks. To evaluate LADEX, we compare generated diagrams with expert ground truths using a trace-based behavioural and an LLM-based matcher. This enables automated measurement of correctness (whether the generated activity diagram includes the ground-truth nodes) and completeness (how many of the ground-truth nodes the generated activity diagram covers). Experiments on two datasets -- a public-domain dataset and an industry dataset from our collaborator, Ciena -- indicate: (1) Both matchers yield similar completeness and correctness comparisons. (2) The critique-refine loop improves structural validity, correctness, and completeness compared to single-pass generation. (3) Activity diagrams refined based on algorithmic structural checks achieve structural consistency, whereas those refined based on LLM-based checks often still show structural inconsistencies.

Figures

Figures reproduced from arXiv: 2509.03463 by the authors.

Figure 1
Figure 1. Motivating example ment in our work since our formalism unifies different UML syntactic elements; for example, we treat forks and merges uniformly as action nodes (Definition 2.1). One rule – namely, the exclusion of dangling transitions – is intrinsically enforced in our generation process, as our encoding (see Section III) does not admit transitions without both source and target nodes. The remaining six constrain… view at source ↗
Figure 2
Figure 2. Example of one execution iteration of LADEX: (a) a candidate activity diagram generated from the process description [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. An overview of the architecture of LADEX. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: A candidate activity diagram generated from the process description in Figure1(a) that is structurally sound based on [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of average correctness and completeness scores between L-Match (x-axis) and B-Match (y-axis): Blue and orange points, respectively, represent the average results of each variant across all runs and LLMs for the PAGED and Industry datasets. Dashed lines indic…
Figure 6
Figure 6. Figure 6: Comparison of the LADEX variants with a refinement [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 8
Figure 8. Figure 8: Critique of Figure 2(a) generated by the LADEX-Alg-LLM variant. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Comparison of the best-performing variants of [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Outlines of the prompts used at each step of LADEX. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Automated Domain Model Extraction from Source Code using Heuristics and Open-Source LLMs

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A locally deployable LLM pipeline with documentation-based ranking and iterative classification extracts domain models from Java code, achieving an F1 of 0.90 for classes but with evaluation caveats.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Extracting domain models from natural-language requirements: approach and industrial evaluation,

    C. Arora, M. Sabetzadeh, L. C. Briand, and F. Zimmer, “Extracting domain models from natural-language requirements: approach and industrial evaluation,” inProceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems, Saint-Malo, France, October 2-7, 2016, B. Baudry and B. Combemale, Eds. ACM, 2016, pp. 250–26...

    work page 2016

  2. [2]

    An active learning approach for improving the accuracy of automated domain model extraction,

    C. Arora, M. Sabetzadeh, S. Nejati, and L. C. Briand, “An active learning approach for improving the accuracy of automated domain model extraction,”ACM Trans. Softw. Eng. Methodol., vol. 28, no. 1, pp. 4:1–4:34, 2019. [Online]. Available: https://doi.org/10.1145/3293454

    work page doi:10.1145/3293454 2019

  3. [3]

    Unified modeling language (UML) version 2.5.1,

    S. Cook, C. Bock, P. Rivett, T. Rutt, E. Seidewitz, B. Selic, and D. Tolbert, “Unified modeling language (UML) version 2.5.1,” Object Management Group (OMG), Standard, Dec. 2017. [Online]. Available: https://www.omg.org/spec/UML/2.5.1

    work page 2017

  4. [4]

    2.0, jan 2011, accessed: 2025-08-21

    OMG,Business Process Model and Notation (BPMN), Version 2.0, Object Management Group Std., Rev. 2.0, jan 2011, accessed: 2025-08-21. [Online]. Available: http://www.omg.org/spec/BPMN/2.0

    work page 2011

  5. [5]

    Automated change impact analysis between sysml models of requirements and design,

    S. Nejati, M. Sabetzadeh, C. Arora, L. C. Briand, and F. Mandoux, “Automated change impact analysis between sysml models of requirements and design,” inProceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016, T. Zimmermann, J. Cleland-Huang, and Z. Su, Eds. ACM, 2...

    work page arXiv 2016

  6. [6]

    The state of practice in model-driven engineering,

    J. Whittle, J. Hutchinson, and M. Rouncefield, “The state of practice in model-driven engineering,”IEEE Software, vol. 31, no. 3, pp. 79–85, 2014

    work page 2014

  7. [7]

    Model generation with llms: From requirements to UML sequence diagrams,

    A. Ferrari, S. Abualhaija, and C. Arora, “Model generation with llms: From requirements to UML sequence diagrams,” in32nd IEEE International Requirements Engineering Conference, RE 2024 - Workshops, Reykjavik, Iceland, June 24-25, 2024. IEEE, 2024, pp. 291–300. [Online]. Available: https://doi.org/10.1109/REW61692.2024. 00044

    arXiv 2024

  8. [8]

    Automating data flow diagram generation from user stories using large language models,

    G. B. Herwanto, “Automating data flow diagram generation from user stories using large language models,” inJoint Proceedings of REFSQ-2024 Workshops, Doctoral Symposium, Posters & Tools Track, and Education and Training Track co-located with the 30th International Conference on Requirements Engineering: Foundation for Software Quality (REFSQ 2024), Winter...

    work page 2024

Show all 36 references
  1. [9]

    Automated derivation of UML sequence diagrams from user stories: Unleashing the power of generative AI vs. a rule- based approach,

    M. Jahan, M. M. Hassan, R. Golpayegani, G. Ranjbaran, C. Roy, B. Roy, and K. A. Schneider, “Automated derivation of UML sequence diagrams from user stories: Unleashing the power of generative AI vs. a rule- based approach,” inProceedings of the ACM/IEEE 27th International Conf...

    2024

  2. [10]

    Multi-step iterative automated domain modeling with large language models,

    Y . Yang, B. Chen, K. Chen, G. Mussbacher, and D. Varró, “Multi-step iterative automated domain modeling with large language models,” in Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, MODELS Companion 2024, Linz, Au...

    2024

  3. [11]

    Consistent graph model generation with large language models,

    B. Chen, “Consistent graph model generation with large language models,” in47th IEEE/ACM International Conference on Software Engineering, ICSE 2025 - Companion Proceedings, Ottawa, ON, Canada, April 27 - May 3, 2025. IEEE, 2025, pp. 218–219. [Online]. Available: https://doi.o...

    2025

  4. [12]

    Addiff: semantic differencing for activity diagrams,

    S. Maoz, J. O. Ringert, and B. Rumpe, “Addiff: semantic differencing for activity diagrams,” inSIGSOFT/FSE’11 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-19) and ESEC’11: 13th European Software Engineering Conference (ESEC-13), Szeged, Hungary, S...

    2011

  5. [13]

    Matching and merging of statecharts specifications,

    S. Nejati, M. Sabetzadeh, M. Chechik, S. M. Easterbrook, and P. Zave, “Matching and merging of statecharts specifications,” in 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007. IEEE Computer Society, 2007, pp. 54–64. [Onl...

    2007 doi

  6. [14]

    Matching and merging of variant feature specifications,

    ——, “Matching and merging of variant feature specifications,”IEEE Transaction on Software Engineering, vol. 38, no. 6, pp. 1355–1375,

  7. [15]

    PAGED: A benchmark for procedural graphs extraction from documents,

    W. Du, W. Liao, H. Liang, and W. Lei, “PAGED: A benchmark for procedural graphs extraction from documents,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, L. Ku...

    2024 doi

  8. [16]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, A. Madry, A. Baker-Whitcomb, A. Beutel, A. Borzunov, A. Carney, and Others, “Gpt-4o system card,” 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2410.21276

    2024 doi arXiv

  9. [17]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, and Others, “Openai o1 system card,”CoRR, vol. abs/2412.16720, 2024. [Online]. Available: https://doi.org/10.48550/arXiv.2412.16720

    2024 doi arXiv

  10. [18]

    Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, and Others, “Deepseek- r1: Incentivizing reasoning capability in llms via reinforcement learning,”CoRR, vol. abs/2501.12948, 2025. [Online]. Available: https://doi.org/10.48550/arX...

    2025 doi arXiv

  11. [19]

    Requirement or not, that is the question: A case from the railway industry,

    S. Bashir, M. Abbas, M. Saadatmand, E. P. Enoiu, M. Bohlin, and P. Lindberg, “Requirement or not, that is the question: A case from the railway industry,” inRequirements Engineering: Foundation for Software Quality - 29th International Working Conference, REFSQ 2023, Barcelona...

    2023 doi

  12. [20]

    Automated demarcation of requirements in textual specifications: a machine learning-based approach,

    S. Abualhaija, C. Arora, M. Sabetzadeh, L. C. Briand, and M. Traynor, “Automated demarcation of requirements in textual specifications: a machine learning-based approach,”Empir. Softw. Eng., vol. 25, no. 6, pp. 5454–5497, 2020. [Online]. Available: https://doi.org/10.1007/s106...

    2020 doi

  13. [21]

    draw.io,

    JGraph, “draw.io,” https://www.draw.io/, 2021, accessed: 2025-08-21

    2021

  14. [22]

    Simulation-based graph similarity,

    O. Sokolsky, S. Kannan, and I. Lee, “Simulation-based graph similarity,” inTools and Algorithms for the Construction and Analysis of Systems, 12th International Conference, TACAS 2006 Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2006...

    2006 doi

  15. [23]

    Large language models sensitivity to the order of options in multiple-choice questions,

    P. Pezeshkpour and E. Hruschka, “Large language models sensitivity to the order of options in multiple-choice questions,” inFindings of the Association for Computational Linguistics: NAACL 2024, Mexico City, Mexico, June 16-21, 2024, K. Duh, H. Gómez-Adorno, and S. Bethard, Ed...

    2024 doi

  16. [24]

    Ollama, “Ollama,” https://github.com/ollama/ollama, 2023, accessed: 2025-08-21

    2023

  17. [25]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” inProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, ...

    2019 doi

  18. [26]

    mgte: Generalized long-context text representation and reranking models for multilingual text retrieval,

    X. Zhang, Y . Zhang, D. Long, W. Xie, Z. Dai, J. Tang, H. Lin, B. Yang, P. Xie, F. Huang, M. Zhang, W. Li, and M. Zhang, “mgte: Generalized long-context text representation and reranking models for multilingual text retrieval,” pp. 1393–1412, 2024. [Online]. Available: https:/...

    2024 doi

  19. [27]

    Towards general text embeddings with multi-stage contrastive learning,

    Z. Li, X. Zhang, Y . Zhang, D. Long, P. Xie, and M. Zhang, “Towards general text embeddings with multi-stage contrastive learning,” 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2308.03281

    2023 doi arXiv

  20. [28]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” inBreak- throughs in statistics: Methodology and distribution. Springer, 1992, pp. 196–202

    1992

  21. [29]

    A critique and improvement of the cl common language effect size statistics of mcgraw and wong,

    A. Vargha and H. D. Delaney, “A critique and improvement of the cl common language effect size statistics of mcgraw and wong,”Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101–132, 2000

    2000

  22. [30]

    Controlling the false discovery rate: a practical and powerful approach to multiple testing,

    Y . Benjamini and Y . Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,”Journal of the Royal statistical society: series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995

    1995

  23. [31]

    The proof and measurement of association between two things,

    C. Spearman, “The proof and measurement of association between two things,”The American Journal of Psychology, vol. 100, no. 3/4, pp. 441–471, 1987. [Online]. Available: https://doi.org/10.2307/1422689

    1987 doi

  24. [32]

    Azure openai reasoning models - gpt-5 series, o3-mini, o1, o1-mini,

    Microsoft, “Azure openai reasoning models - gpt-5 series, o3-mini, o1, o1-mini,” 2025, accessed: 2025-08-21. [Online]. Available: https: //learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/reasoning

    2025

  25. [33]

    Chatbot arena: An open platform for evaluating llms by human preference,

    W. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. I. Jordan, J. E. Gonzalez, and I. Stoica, “Chatbot arena: An open platform for evaluating llms by human preference,” 2024. [Online]. Available: https://openreview.net/forum?id=3MW8GKNyzI

    2024

  26. [34]

    Alphatrans: A neuro-symbolic compositional approach for repository-level code translation and validation,

    A. R. Ibrahimzada, K. Ke, M. Pawagi, M. S. Abid, R. Pan, S. Sinha, and R. Jabbarvand, “Alphatrans: A neuro-symbolic compositional approach for repository-level code translation and validation,”Proc. ACM Softw. Eng., vol. 2, no. FSE, pp. 2454–2476, 2025. [Online]. Available: ht...

    2025 doi

  27. [35]

    Neurosymbolic architectural reasoning: Towards formal analysis through neural software architecture inference,

    S. Herbold, C. Knieke, A. Rausch, and C. Schindler, “Neurosymbolic architectural reasoning: Towards formal analysis through neural software architecture inference,” in1st IEEE/ACM International Workshop on Neuro-Symbolic Software Engineering, NSE@ICSE 2025, Ottawa, ON, Canada,...

    2025

  28. [2012]

    Available: https://doi.org/10.1109/TSE.2011.112

    [Online]. Available: https://doi.org/10.1109/TSE.2011.112

    2011 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.