Pith. sign in

REVIEW 2 major objections 4 minor 64 references

Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement

T0 review · 2 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Putting a production SysMLv2 conformance checker inside the generation loop, with deterministic error diagnostics fed back until zero errors remain, lifts production acceptance from 51.16% to 100% across 604 prompt–model cases.

desk verdict Useful benchmark-scale result and a valuable trajectory corpus; the 100% acceptance endpoint is solid, but the paper oversells diagnostics as the mechanism without a no-feedback control. read the letter →

arxiv 2607.14162 v1 pith:JCXKYRJE submitted 2026-07-14 cs.SE

Chance LaVoie , Eladio Andujar Lugo , Taylan G. Topcu , Levent Burak Kara This is my paper · ORCID
classification cs.SE
keywords model-basedsystemsengineeringSysMLv2generationnaturallanguagetomodelconformancecheckinggenerate-check-repairloopLLMiterativerefinementgrammarvsproduction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models asked to turn natural-language system descriptions into SysMLv2 artifacts fail industrial conformance checks more often than they pass: in single-shot generation, only 309 of 604 prompt–model cases were accepted. The paper claims that placing a production SysMLv2 conformance checker inside a generate–check–repair loop — feeding its deterministic error diagnostics back into the model until zero errors remain — makes every case pass, with most resolved in one repair cycle. The point is that production acceptance, not grammar-level parsing, is the criterion that matters for real engineering use, and the checker already sits in the toolchain. The paper also shows grammar parsing overstates deployability, with 60 artifacts parsing cleanly yet failing production checks.

What carries the argument

The load-bearing mechanism is the generate–check–repair loop, in which the candidate SysMLv2 model is checked by a production conformance checker and its deterministic diagnostics are inserted into the next prompt verbatim; the loop terminates only when the checker reports zero errors. The checker thus acts as an oracle that enforces the same model-wide constraints — name resolution, typing, ownership, multiplicity — that industrial modeling environments enforce, shifting the acceptance criterion from grammar plausibility to tool-level usability.

What would settle it

Run the same 604 prompt–model cases with the same retry budget but strip the diagnostics from the repair prompt, asking the model simply to try again. If acceptance after one retry in that no-feedback arm reaches roughly the observed 84% (or cumulative acceptance matches across the full budget), the checker feedback is not the causal driver. A smaller probe: on the 295 cases that fail at first attempt, compare acceptance at the next attempt with and without diagnostics.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that conformance-driven iterative refinement converts probabilistic LLM outputs into production-accepted SysMLv2 artifacts. Across 151 prompts and four model backends (604 prompt–model cases), single-shot production conformance was 51.16%, while the conformance-gated pipeline achieved 100.00%. Acceptance is defined as zero errors from the production checker — the same check an engineer hits when loading a model into an industrial tool. Convergence is front-loaded: 84.44% accepted after one repair cycle, T90 = 2, T95 = 3, T99 = 4, with median total attempts 1 and maximum 8. The paper interprets this as deterministic diagnostics providing a stabl

Load-bearing premise

The load-bearing premise is that the deterministic conformance diagnostics — not merely the extra generation attempts — are what drive the convergence; the paired design compares first and final candidates of the same trajectory without a no-feedback control.

Editorial extensions

If this is right

  • If correct, the approach removes the structural-syntax barrier to LLM-assisted SysMLv2 authoring, making generated models loadable and usable in industrial tools without manual correction.
  • Reliability moves from the model provider to a shared deterministic backend: acceptance no longer depends on first-attempt luck, and the same loop works across different LLM backends.
  • Grammar validity is shown to be necessary but not sufficient, so downstream validation should target production conformance rather than parsing alone.
  • The quick convergence pattern (median one attempt, 99.67% accepted by cycle 4) suggests most repair cases involve shallow structural fixes rather than unstable oscillations.
  • The released trajectory corpus of 1,043 iteration artifacts gives other researchers a resource for studying repair dynamics and training on intermediate states.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be a control condition that regenerates candidates the same number of times without feeding back diagnostics; that would isolate how much of the convergence is due to the feedback signal versus resampling alone.
  • If the pattern holds, the same oracle-guided loop could transfer to other formal modeling languages or schemas that have a deterministic conformance checker, turning an existing validator into a control mechanism during generation.
  • Once structural conformance is guaranteed, semantic checks (requirement traceability, behavior validation) could be layered on top, moving from 'loadable' to 'correct' — a direction the paper explicitly leaves open.
  • The observed early-cycle contraction factor (roughly one-third residual reduction per cycle) points toward a geometric convergence structure; formalizing when diagnostics guarantee such contraction is an open mathematical question the paper does not answer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a generate–check–repair loop for natural-language–to–SysMLv2 translation. A production conformance checker (SysIDE) is used as the oracle: each candidate model is checked, the deterministic diagnostics are fed back as repair prompts, and the loop terminates only when the checker reports zero errors. The method is evaluated on the full SysMBench prompt set (151 prompts) with four LLM backends, yielding 604 prompt–model cases. The authors report single-shot production conformance of 51.16% (309/604) and pipeline conformance of 100.00% (604/604), with a median of 1 total attempt, a maximum of 8 attempts, and time-to-threshold T90=2, T95=3, T99=4. A Clopper–Pearson one-sided 95% lower bound on convergence probability is computed as 0.9951. The paper also releases a trajectory-level corpus of 1,043 iteration artifacts and a 10-case demonstration that grammar-level (ANTLR) parsing does not imply production conformance.

Significance. If the causal interpretation is supported, the paper would provide a practical reliability layer for LLM-generated SysMLv2: industrial-tool acceptance can be enforced by embedding a production checker inside the generation loop, and the released trajectory corpus would be a valuable resource for studying repair dynamics. The empirical gap between grammar parsing and production conformance (60/369 parseable artifacts fail production checks) is a useful quantitative contribution. The reported statistics are internally consistent, the mathematical derivations check out, and the code/dataset release supports reproducibility. However, the central 'control mechanism' claim currently rests on an untested comparison against independent resampling, and the headline 100% conformance is an artifact of the termination condition.

major comments (2)
  1. [§3.1, §4.3, abstract] The causal claim that deterministic conformance diagnostics, rather than the extra sampling iterations, drive convergence is not supported by the experimental design. §3.1 pairs the initial candidate (k=0) with the final candidate of the same trajectory, which conflates feedback with a larger sampling budget. With the observed single-shot pass rate p=0.5116, independent resampling alone predicts A_1≈76.2% and A_8≈99.6%; for the weakest backend (p≈0.39), A_8≈98.2%. The observed A_1=84.44% is higher than the resampling prediction, suggesting feedback helps, but no control arm measures a no-feedback trajectory with the same attempt budget. Consequently, the claims in §4.3 that 'deterministic conformance diagnostics drive the dominant convergence dynamics' and in the abstract that the method acts as a 'control mechanism' are interpretations, not tested conclusions. Appendix C's finding that
  2. [§3.2, §4.1, abstract] The headline '100.00% conformance' is an artifact of the termination condition. §3.2 states 'A run terminates only when the checker reports zero errors,' so the final artifact of every run is conformant by construction; the statistic '604/604 (100.00%)' carries no information about the method's effectiveness. The informative empirical results are the repair-cycle distribution (median 1 total attempt, T90=2, T95=3, T99=4) and the single-shot baseline of 51.16%. I recommend reframing the abstract and §4.1 to present the repair-cycle distribution and the comparison to single-shot as the primary outcomes, and to state explicitly that 100% is the loop's invariant, not an empirically discovered success rate.
minor comments (4)
  1. [§4.4] The Clopper–Pearson bound treats the 604 prompt–model cases as independent Bernoulli trials. Because the cases are structured as 151 prompts × 4 model backends, outcomes for the same prompt may be positively correlated (e.g., prompt difficulty), which would reduce the effective sample size and make the 95% bound anti-conservative. A cluster bootstrap by prompt or a mixed-effects model would provide a more defensible interval. The bound would likely remain high, but the reported '95% confidence' should be calibrated to the actual sampling structure.
  2. [§3.4.2] The contraction analysis uses only three ratios (ρ0–ρ2) to infer an 'approximately multiplicative' pattern. The paper is careful to call this descriptive, but the phrase 'early cycles remove roughly two-thirds of the remaining failures per cycle' may overstate the stability of the estimate given the small number of residual points. Consider reporting the ratios with their uncertainty or explicitly noting the small-sample caveat.
  3. [§4.6 / Appendix B] The statement 'no meaningful relationship between iterations-to-success and either SysMBench difficulty or the length of the generated SysML output' is drawn from fits with R²=0.183 (difficulty) and R²=0.0011 (length). While the length fit is essentially flat, the difficulty fit shows a modest trend (pooled means 1.766, 1.668, 1.938, 1.667, 1.583 across difficulty levels). The wording 'no meaningful relationship' is a bit strong; consider 'weak or inconsistent relationship'.
  4. [§4.3 / Table 1] The tail bucket 'k=5–7' is not broken out; the maximum repair cycle is 7 (attempts 8). Reporting the exact counts for k=5, 6, 7 would improve transparency, especially since the aggregate 2 cases drive the 'maximum 8' claim.

Circularity Check

1 steps flagged · score 5.0 of 10

The 100% conformance result is the loop's exit condition by construction; the bounded-termination data are the non-circular empirical core.

  1. self definitional [§3.2 (Methodology) and §4.1/Abstract]
    "A run terminates only when the checker reports zero errors. ... With the conformance-checker-in-the-loop, the pipeline outputs conformed for 604/604 cases (100.00%)."

    The pipeline conformance metric counts runs whose final candidate has zero SysIDE errors. The loop is defined to continue until the checker reports zero errors, so for any run that terminates, success is guaranteed by the stopping rule. The abstract's 'our approach achieves 100.00% conformance' therefore presents a definitional property of the loop (conditional on termination) as the empirical outcome. The genuinely empirical content—that all 604 runs did terminate, with median 1 attempt and max 8—does not reduce to the definition, but the headline 100% rate itself is fixed by construction.

full rationale

The only defensible circular step is the headline conformance rate: it is the termination condition of the generate–check–repair loop. The paper is transparent about this ('A run terminates only when the checker reports zero errors'), and it also reports non-circular iteration-to-success statistics (mean 1.727, median 1, max 8; T90=2, T95=3, T99=4) and a useful ANTLR-vs-SysIDE gap analysis. The causal claim that 'deterministic conformance diagnostics drive the dominant convergence dynamics' (§4.3) is under-supported because the paired design lacks a no-feedback resampling control; Appendix C even shows 64.6% of persistent exact-error transitions are 'unaddressed and not fixed.' However, a missing control is an experimental validity concern, not circularity: the data would still be informative as bounded-termination evidence. Self-citations ([15]) are to the authors' own repository, but they are used only for auxiliary demonstrations and data release, not as the proof of the central claim. No uniqueness theorem, imported ansatz, or renamed known result is load-bearing. Accordingly, the circularity score is moderate rather than high: one headline result reduces by construction, while the central empirical content (bounded termination and conformance-gap measurements) remains independent.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted: contraction ratios, T90/T95/T99, and Clopper–Pearson bounds are descriptive statistics of observed trajectories, honestly labeled as such. No new physical or conceptual entities are postulated; the framework composes existing components (LLM, SysIDE checker, a Python controller). The axioms are domain assumptions about the oracle, the benchmark, the grammar-checker operationalization, and the treatment of stochastic LLM outputs. The main unstated load is the assumption that the correctness signal (diagnostics) is the causal driver of convergence, which is an empirical premise tested nowhere as a control.

assumptions (5)
  • domain assumption SysIDE zero-error acceptance is a valid proxy for production-level usability of a SysMLv2 model in industrial environments.
    Central to the framework's framing (§1, §3.2). The paper explicitly limits the claim to a single conformance backend and notes different toolchains may enforce additional constraints (§7).
  • domain assumption The SysMBench 151-prompt set is representative of natural-language-to-SysMLv2 specification tasks, at least for structural (package/part/port/connection) model content.
    Evaluation universe is the full SysMBench prompt set (§3.3). §5 and §7 explicitly narrow the claims to SysMBench-style prompts and to structural content, acknowledging requirement/parametric syntax is barely represented.
  • domain assumption ANTLR parse success is a fair operationalization of 'grammar validity', and grammar validity is necessary for production conformance.
    Used to build the 2×2 grammar-versus-production analysis (§3.4.4). The 'necessary' direction is supported by 0/604 observed cases of SysIDE-pass/ANTLR-fail (§4.5), but the parser is a specific implementation (Sireum/hamr-sysml-parser [51]).
  • domain assumption One sampled trajectory per prompt–model pair is treated as representative of backend behavior; LLM outputs are compared as if deterministic for the paired design.
    The paired design (§3.1) uses the single initial candidate per prompt–model pair as the baseline. Sampling parameters (temperature, seed) are not reported in the paper.
  • standard math Clopper–Pearson exact binomial inference and the descriptive contraction-ratio computation are valid for the all-success case.
    §3.4.3; computation verified: (0.05)^(1/604)≈0.9951. The contraction ratios are descriptive summaries of the 604 observed trajectories, and the paper disclaims formal convergence guarantees.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement." pith.science (2026). https://pith.science/paper/JCXKYRJE

@misc{pith2026260714162,
  author       = {Pith},
  title        = {Pith review of: Natural-Language to SysMLv2 Translation via Conformance-Driven Iterative Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCXKYRJE}},
  note         = {Machine review of arXiv:2607.14162}
}
read the original abstract

Model-Based Systems Engineering (MBSE) relies on formal system models as primary technical artifacts for representing requirements, structure, and behavior across the system lifecycle. With the standardization of SysMLv2 as a textual language, interest is increasing in translating natural-language descriptions directly into executable models. For practical deployment, generated models must be accepted by industrial modeling environments, not merely satisfy grammar constraints. We present a conformance-checker-driven framework for reliable natural-language-to-SysMLv2 translation that enforces production-level acceptance as the termination condition. The system embeds a SysMLv2 conformance checker within a generate-check-repair loop. Each model is evaluated using the checker, and deterministic diagnostics are incorporated into revisions until zero conformance errors are achieved. Using the production checker as the oracle ensures the framework targets deployability rather than grammar plausibility. We evaluate the approach on the full SysMBench prompt set of 151 prompts across four large language model backends, yielding 604 prompt-model cases. Single-shot generation achieves 51.16% production-conformance acceptance, while our approach achieves 100.00% conformance. By elevating production conformance from a post-processing check to a control mechanism within generation, the framework converts probabilistic outputs into production-accepted SysMLv2 artifacts suitable for loading, visualization, and engineering use.

Figures

Figures reproduced from arXiv: 2607.14162 by the authors.

Figure 1
Figure 1. Conformance-checker-in-the-loop workflow for natural-language–to–SysMLv2 generation. A commercial [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Representative SysMBench prompt excerpts illustrating the variety and structure of the benchmark inputs. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Overall single-shot vs. final pipeline production conformance across all 604 prompt-level cases. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Per-model single-shot vs. final pipeline production conformance. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Cumulative production conformance versus repair cycles ( [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Per-model cumulative production conformance versus repair cycles ( [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Initial-candidate grammar (ANTLR) vs. production conformance (SysIDE), [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Difficulty-conditioned repair effort with model split and pooled trend. The pooled mean iterations-to-success [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Generated SysML output length analysis. The left panel shows a scatter plot of generated SysML line count [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Persistent exact-error outcomes for the 240 transitions where the same conformance error [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 9 canonical work pages

  1. [1]

    Jeff A. Estefan. Survey of model-based systems engineering (mbse) methodologies. Technical Report Rev. B, International Council on Systems Engineering (INCOSE), Seattle, WA, USA, May 2008. URL https: //www.omg.org/sysml/MBSE_Methodology_Survey_RevB.pdf. Prepared for the INCOSE MBSE Initiative

    2008

  2. [2]

    Madni and Michael Sievers

    Azad M. Madni and Michael Sievers. Model-based systems engineering: Motivation, current status, and research opportunities.Systems Engineering, 21(3):172–190, May 2018. doi: 10.1002/sys.21438

    doi:10.1002/sys.21438 2018

  3. [3]

    Systems engineering vision 2035: Engineering solutions for a better world

    International Council on Systems Engineering (INCOSE). Systems engineering vision 2035: Engineering solutions for a better world. Technical report, INCOSE, 2021. URL https://www.incose.org/docs/ default-source/aboutse/se-vision-2035.pdf

  4. [4]

    OMG Systems Modeling Language (SysML)

    Object Management Group. OMG Systems Modeling Language (SysML). Technical Report Version 2.0, Object Management Group, September 2025. URLhttps://www.omg.org/spec/SysML/2.0/About-SysML. Formal specification

    2025

  5. [5]

    Value and benefits of model-based systems engineering (mbse): Evidence from the literature.Systems Engineering, 24(1):51–66, 2021

    Kaitlin Henderson and Alejandro Salado. Value and benefits of model-based systems engineering (mbse): Evidence from the literature.Systems Engineering, 24(1):51–66, 2021

    2021

  6. [6]

    Campo, Thomas Teper, Casey E

    Kelly X. Campo, Thomas Teper, Casey E. Eaton, Anna M. Shipman, Garima Bhatia, and Bryan Mesmer. Model- based systems engineering: Evaluating perceived value, metrics, and evidence through literature.Systems Engineering, 26(1):104–129, 2023. doi: 10.1002/sys.21644

    doi:10.1002/sys.21644 2023

  7. [7]

    Ai4se and se4ai: A research roadmap.Insight, 23(1):8–14, 2020

    Tom McDermott, Dan DeLaurentis, Peter Beling, Mark Blackburn, and Mary Bone. Ai4se and se4ai: A research roadmap.Insight, 23(1):8–14, 2020

    2020

  8. [8]

    Mohammed Husain, Paul Wach, and Taylan G. Topcu. Can Large Language Models Accelerate Digi- tal Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empiri- cal Exploration. InConference on Systems Engineering Research, pages 371–385. Springer, 2024. URL https://link.springer.com/chapter/10.1007/978-3-031-62554-1_23

    work page doi:10.1007/978-3-031-62554-1_23 2024

Show all 64 references
  1. [9]

    The impact of ai on developer productivity: Evidence from github copilot, February 2023

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer productivity: Evidence from github copilot, February 2023

    2023

  2. [10]

    The effects of generative ai on high-skilled work: Evidence from three field experiments with software devel- opers.Working Paper, 2025

    Kevin Zheyuan Cui, Sonia Jaffe, Mert Demirer, Sida Peng, Alexi Quintana, Leon Musolff, and Tobias Salz. The effects of generative ai on high-skilled work: Evidence from three field experiments with software devel- opers.Working Paper, 2025. URL https://economics.mit.edu/sites/...

    2025

  3. [11]

    The productivity effects of generative ai: Evidence from a field experiment with github copilot, March 2024

    Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. The productivity effects of generative ai: Evidence from a field experiment with github copilot, March 2024. MIT Generative AI Working Paper

    2024

  4. [12]

    Systemp: A multi-agent system for template-based generation of sysml v2, June 2025

    Yasmine Bouamra, Bruno Yun, Alexandre Poisson, and Frédéric Armetta. Systemp: A multi-agent system for template-based generation of sysml v2, June 2025

    2025

  5. [13]

    An agent-based approach for the automatic generation of valid sysmlv2 models in industrial contexts.Computers in Industry, 172:104350, November 2025

    Eduardo Cibrián, Jose Olivert-Iserte, Juan Llorens, and Jose María Álvarez-Rodríguez. An agent-based approach for the automatic generation of valid sysmlv2 models in industrial contexts.Computers in Industry, 172:104350, November 2025. doi: 10.1016/j.compind.2025.104350

    2025

  6. [14]

    A system model generation benchmark from natural language requirements.arXiv preprint arXiv:2508.03215, August 2025

    Dongming Jin, Zhi Jin, Linyu Li, Zheng Fang, Jia Li, Xiaohong Chen, and Yixing Luo. A system model generation benchmark from natural language requirements.arXiv preprint arXiv:2508.03215, August 2025. doi: 10.48550/arXiv.2508.03215

    2025 doi arXiv

  7. [15]

    Natural-language-to-sysmlv2- translation-via-conformance-driven-iterative-refinement

    Chance LaV oie, Eladio Andujar Lugo, and Levent Burak Kara. Natural-language-to-sysmlv2- translation-via-conformance-driven-iterative-refinement. https://github.com/cmuchancel/ NL-to-SysMLv2-via-Conformance-Driven-Refinement , 2026. Code, artifacts, and analysis scripts for th...

    2026

  8. [16]

    DoD Digital Engineering Strategy

    Office of the Deputy Assistant Secretary of Defense for Systems Engineering. DoD Digital Engineering Strategy. Technical report, U.S. Department of Defense, Washington, DC, 2018. URL https://ac.cto.mil/wp-content/uploads/2019/06/2018-Digital-Engineering-Strategy_ Approved_Prin...

    2018

  9. [17]

    DoD Instruction 5000.89: Test and Evaluation

    Office of the Under Secretary of Defense for Research and Engineering. DoD Instruction 5000.89: Test and Evaluation. Technical Report DoDI 5000.89, U.S. Department of Defense, Washington, DC, November 2020

    2020

  10. [18]

    Digital engineering transformation across the Department of Defense.The Journal of Defense Modeling and Simulation, 16(4):325–338, 2019

    Phil Zimmerman, Tracee Gilbert, and Frank Salvatore. Digital engineering transformation across the Department of Defense.The Journal of Defense Modeling and Simulation, 16(4):325–338, 2019. 14 Natural-Language to SysMLv2 Translation

    2019

  11. [19]

    Topcu and Zoe Szajnfarber

    Taylan G. Topcu and Zoe Szajnfarber. Navigating the golden triangle: The need to jointly consider modularization and interface choices when making performance, cost, and schedule tradeoffs for complex system development. Systems Engineering, 28(3):310–324, 2025. doi: 10.1002/sys.21796

    2025 doi

  12. [20]

    Trust in an ai versus a human teammate: The effects of teammate identity and performance on human-ai cooperation.Computers in Human Behavior, 139: 107536, 2023

    Guanglu Zhang, Leah Chong, Kenneth Kotovsky, and Jonathan Cagan. Trust in an ai versus a human teammate: The effects of teammate identity and performance on human-ai cooperation.Computers in Human Behavior, 139: 107536, 2023

    2023

  13. [21]

    Topcu, Mohammed Husain, Max Ofsa, and Paul Wach

    Taylan G. Topcu, Mohammed Husain, Max Ofsa, and Paul Wach. Trust at your own peril: A mixed methods exploration of the ability of large language models to generate expert-like systems engineering artifacts and a characterization of failure modes.Systems Engineering, February 2...

    2025 doi

  14. [22]

    Reconsidering the trade-off between expertise and flexibility: a cognitive entrenchment perspective.The Academy of Management Review, 35(4):579–603, 2010

    Erik Dane. Reconsidering the trade-off between expertise and flexibility: a cognitive entrenchment perspective.The Academy of Management Review, 35(4):579–603, 2010. URLhttps://www.jstor.org/stable/29765006

    2010

  15. [23]

    How design engineers spend their time: Job content and task satisfaction.Design studies, 33 (4):391–425, 2012

    Mark A Robinson. How design engineers spend their time: Job content and task satisfaction.Design studies, 33 (4):391–425, 2012

    2012

  16. [24]

    Towards developing metrics to evaluate digital engineering.Systems Engineering, 26(1):3–31, 2023

    Kaitlin Henderson, Tom McDermott, Eileen Van Aken, and Alejandro Salado. Towards developing metrics to evaluate digital engineering.Systems Engineering, 26(1):3–31, 2023

    2023

  17. [25]

    SysIDE: SysML v2 textual editing and analysis system: Overview and applications.CEAS Space Journal, February 2025

    Juozas Vaicenaviˇcius, Tilo Wiklund, Daumantas Kavolis, Simonas Draukšas, Antanas Kalkauskas, and Rimantas Vaicenaviˇcius. SysIDE: SysML v2 textual editing and analysis system: Overview and applications.CEAS Space Journal, February 2025. doi: 10.1007/s12567-025-00595-x

    2025 doi

  18. [26]

    Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith

    Edmund M. Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. Counterexample-guided abstraction refinement. In E. Allen Emerson and Aravinda Prasad Sistla, editors,Proceedings of the 12th International Conference on Computer Aided Verification (CAV 2000), pages 154–1...

    doi

  19. [27]

    PhD thesis, University of California, Berkeley, Decem- ber 2008

    Armando Solar-Lezama.Program Synthesis by Sketching. PhD thesis, University of California, Berkeley, Decem- ber 2008. URLhttps://www2.eecs.berkeley.edu/Pubs/TechRpts/2008/EECS-2008-176.html

    2008

  20. [28]

    Seshia, and Ashish Tiwari

    Susmit Jha, Sumit Gulwani, Sanjit A. Seshia, and Ashish Tiwari. Oracle-guided component-based program synthesis. InInternational Conference on Software Engineering (ICSE), pages 215–224, 2010. doi: 10.1145/ 1806799.1806833

    2010

  21. [29]

    Rajeev Alur, Rastislav Bodik, Garvit Juniwal, Milo M. K. Martin, Mukund Raghothaman, Sanjit A. Seshia, Rishabh Singh, Armando Solar-Lezama, Emina Torlak, and Abhishek Udupa. Syntax-guided synthesis. In Proceedings of the 2013 Formal Methods in Computer-Aided Design (FMCAD), pa...

    2013

  22. [30]

    Compilable neural code generation with compiler feedback

    Xin Wang, Wenhu Chen, Xinyun Chen, and William Yang Wang. Compilable neural code generation with compiler feedback. InFindings of the Association for Computational Linguistics: ACL 2022, pages 138–150,

    2022

  23. [31]

    Compiler generated feedback for large language models.arXiv preprint arXiv:2403.14714, March 2024

    Dejan Grubisic, Chris Cummins, V olker Seeker, and Hugh Leather. Compiler generated feedback for large language models.arXiv preprint arXiv:2403.14714, March 2024. doi: 10.48550/arXiv.2403.14714

    2024 doi arXiv

  24. [32]

    Generating sysml behavior models via large language models: An empirical study

    Yuan Wang, Ning Ge, Jiangxi Liu, Zhilong Cao, Zheping Chen, and Chunming Hu. Generating sysml behavior models via large language models: An empirical study. InProceedings of the 16th International Conference on Internetware (Internetware 2025). ACM, 2025. doi: 10.1145/3755881.3755926

    2025

  25. [33]

    John K. DeHart. Leveraging large language models for direct interaction with sysml v2.INCOSE International Symposium, 34(1):2168–2185, 2024. doi: 10.1002/iis2.13262

    2024 doi

  26. [34]

    Andres Arellano, Edward Zontek-Carney, and Mark A. Austin. Frameworks for natural language processing of textual requirements. InProceedings of the IARIA Conference on Advances in Engineering, Science and Management. IARIA, 2015. URL https://www.terpconnect.umd.edu/~austin/enc...

    2015

  27. [35]

    Alaa Abdalazeim and Farid Meziane. Extending ontology-driven natural language generation for requirements engineering using ontouml: A review.Journal of Computer Science and Software Development, 4:1–11, October 2025

    2025

  28. [36]

    Intelligent requirements engineering from natural language and their chaining toward cad models.arXiv preprint arXiv:2007.07825, 2020

    Alain-Jérôme Fougères and Egon Ostrosi. Intelligent requirements engineering from natural language and their chaining toward cad models.arXiv preprint arXiv:2007.07825, 2020. doi: 10.48550/arXiv.2007.07825

    2007 doi arXiv

  29. [37]

    Ontology-based nlp tool for tracing software requirements and conceptual models: An empirical study.Requirements Engineering, 2025

    David Mosquera, Marcela Ruiz, and Oscar Pastor. Ontology-based nlp tool for tracing software requirements and conceptual models: An empirical study.Requirements Engineering, 2025. doi: 10.1007/s00766-025-00447-4. 15 Natural-Language to SysMLv2 Translation

    2025 doi

  30. [38]

    An automated approach to generate sysml models from restricted natural language requirements in chinese.Journal of Computer Research and Development, 58(4):706–730, 2021

    Bao Yang, Zhibin Yang, Yongqiang Yang, Jian Xie, Yong Zhou, Tao Yue, Zhiqiu Huang, and Peng Guo. An automated approach to generate sysml models from restricted natural language requirements in chinese.Journal of Computer Research and Development, 58(4):706–730, 2021. doi: 10.7...

    2021

  31. [39]

    Natural language processing for systems engineering: Automatic generation of systems modelling language diagrams.Knowledge-Based Systems, 259:110071, 2023

    Shaohong Zhong, Andrea Scarinci, and Alice Cicirello. Natural language processing for systems engineering: Automatic generation of systems modelling language diagrams.Knowledge-Based Systems, 259:110071, 2023. doi: 10.1016/j.knosys.2022.110071

    2023

  32. [40]

    Text-to-model transformation: Natural language-based model generation framework.Systems, 12(9):369, 2024

    Aditya Akundi, Joshua Ontiveros, and Sergio Luna. Text-to-model transformation: Natural language-based model generation framework.Systems, 12(9):369, 2024. doi: 10.3390/systems12090369

    2024 doi

  33. [41]

    Llm-assisted semantic alignment and integration in collaborative model-based systems engineering using sysml v2

    Zirui Li, Stephan Husung, and Haoze Wang. Llm-assisted semantic alignment and integration in collaborative model-based systems engineering using sysml v2. In2025 IEEE International Symposium on Systems Engineering (ISSE), pages 1–8. IEEE, October 2025. doi: 10.1109/ISSE65546.2...

    2025

  34. [42]

    Matthew Anderson Hendricks and Alice Cicirello. Text to model via SysML: Automated generation of dynamical system computational models from unstructured natural language text via enhanced System Modeling Language diagrams.arXiv preprint arXiv:2507.06803, 2025. URLhttps://arxiv...

    2025 arXiv

  35. [43]

    Grammar-constrained decoding for structured nlp tasks without finetuning

    Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained decoding for structured nlp tasks without finetuning. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URLhttps://aclanthology.org/2023.emn...

    2023

  36. [44]

    Grammar- aligned decoding

    Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, and Loris D’Antoni. Grammar- aligned decoding. InAdvances in Neural Information Processing Systems (NeurIPS 2024), 2024. URL https: //arxiv.org/abs/2405.21047

    2024

  37. [45]

    Grammar-constrained decoding for structured information extraction with low-resource transformers.Frontiers in Artificial Intelligence, 7, 2024

    Carlos Garcia, Isabel Castro, and Mark Johnson. Grammar-constrained decoding for structured information extraction with low-resource transformers.Frontiers in Artificial Intelligence, 7, 2024. doi: 10.3389/frai.2024. 1406857

    2024 doi

  38. [46]

    Jabrw and Qusay I

    Sardar K. Jabrw and Qusay I. Sarhan. A systematic survey on large language models for code generation.ARO: The Scientific Journal of Koya University, 13(2):83–99, 2025. doi: 10.14500/aro.12159

    2025 doi

  39. [47]

    Llmloop: Improving llm-generated code and tests through automated iterative feedback loops

    Ravin Ravi, Dylan Bradshaw, Stefano Ruberto, Gunel Jahangirova, and Valerio Terragni. Llmloop: Improving llm-generated code and tests through automated iterative feedback loops. InIEEE International Conference on Software Maintenance and Evolution (ICSME 2025). IEEE, 2025. URL...

    2025

  40. [48]

    Vecogen: Automating generation of formally verified c code with large language models.arXiv preprint arXiv:2411.19275, 2024

    Merlijn Sevenhuijsen, Khashayar Etemadi, and Mattias Nyberg. Vecogen: Automating generation of formally verified c code with large language models.arXiv preprint arXiv:2411.19275, 2024. doi: 10.48550/arXiv.2411. 19275

    2024 doi arXiv

  41. [49]

    Orvalho, Mikoláš Janota, and Vasco Manquinho

    Pedro M. Orvalho, Mikoláš Janota, and Vasco Manquinho. Counterexample guided program repair using zero-shot learning and maxsat-based fault localization. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI 2025), 2025. URLhttps://arxiv.org/abs/2502.07786

    2025 arXiv

  42. [50]

    Byrd, Robert Zinkov, and Nada Amin

    David Brandfonbrener, Simon Henniger, Sibi Raja, Tarun Prasad, Chloe Loughridge, Federico Cassano, Sabrina Ruixin Hu, Jianang Yang, William E. Byrd, Robert Zinkov, and Nada Amin. VerMCTS: Synthe- sizing multi-step programs using a verifier, a large language model, and tree sea...

    2024 arXiv

  43. [51]

    Sysml v2 parser for hamr

    Sireum Project. Sysml v2 parser for hamr. https://github.com/sireum/hamr-sysml-parser , 2026. GitHub repository sireum/hamr-sysml-parser; commit d7c87942ca9f84de611415c8cca0c5916cc9ccae; Accessed February 22, 2026

    2026

  44. [52]

    Sysml v2 pilot implementation

    Systems Modeling Community. Sysml v2 pilot implementation. https: //github.com/Systems-Modeling/SysML-v2-Pilot-Implementation , 2026. GitHub repository Systems-Modeling/SysML-v2-Pilot-Implementation; commit a5a602d28c570cfe1bb191041a1125f64d917a36; Accessed February 22, 2026

    2026

  45. [53]

    GPT-5.2 Codex Model

    OpenAI. GPT-5.2 Codex Model. https://platform.openai.com/docs/models, 2025. OpenAI Platform Documentation, accessed March 13, 2026

    2025

  46. [54]

    Claude Sonnet 4.6 (claude-sonnet-4-6)

    Anthropic. Claude Sonnet 4.6 (claude-sonnet-4-6). https://docs.anthropic.com/en/docs/ models-overview, 2026. Anthropic API documentation, accessed February 20, 2026

    2026

  47. [55]

    DeepSeek Reasoner v3.2 (deepseek-reasoner)

    DeepSeek AI. DeepSeek Reasoner v3.2 (deepseek-reasoner). https://api-docs.deepseek.com, 2025. DeepSeek API documentation, accessed March 13, 2026. 16 Natural-Language to SysMLv2 Translation

    2025

  48. [56]

    Mistral Large 2512 (mistral-large-2512)

    Mistral AI. Mistral Large 2512 (mistral-large-2512). https://docs.mistral.ai/models, 2025. Mistral AI documentation, accessed March 13, 2026

    2025

  49. [57]

    Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales

    Stefan Banach. Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales. Fundamenta Mathematicae, 3(1):133–181, 1922. doi: 10.4064/fm-3-1-133-181

    1922 doi

  50. [58]

    Wright.Numerical Optimization

    Jorge Nocedal and Stephen J. Wright.Numerical Optimization. Springer, New York, NY , USA, 2 edition, 2006. doi: 10.1007/978-0-387-40065-5

    2006 doi

  51. [59]

    Average convergence rate of evolutionary algorithms.IEEE Transactions on Evolutionary Computation, 20(2):316–321, 2016

    Jiawei He and Guangming Lin. Average convergence rate of evolutionary algorithms.IEEE Transactions on Evolutionary Computation, 20(2):316–321, 2016. doi: 10.1109/TEVC.2015.2444793

    2016

  52. [60]

    Polyak.Introduction to Optimization

    Boris T. Polyak.Introduction to Optimization. Optimization Software, Inc., New York, NY , USA, 1987

    1987

  53. [61]

    C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4):404–413, 1934. doi: 10.1093/biomet/26.4.404. A Auxiliary Demonstration: Grammar Parsability vs. Production Conformance To support the design cho...

    1934 doi

  54. [63]

    ANTLR parse check using the SysMLv2 ANTLR4 parser (generated grammar derived from the SysML v2 Pilot Implementation) [51, 52]

  55. [64]

    unaddressed and not fixed

    Production conformance check using SysIDE. Results were unambiguous: all 10/10 examples passed ANTLR parsing, while 0/10 were accepted by production conformance (10/10 mismatch cases). Conformance-checker diagnostics were domi- nated by unresolved-reference failures (9/10, ref...

  56. [2022]

    URLhttps://aclanthology.org/2022.findings-acl.2/

    2022

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.