{"id":"6c2a948c-bd29-4090-95f4-b843d2dcc9f8","arxiv_id":"2506.21608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-agent LLM pipeline with a rule-based template generator and parser feedback improves syntactic correctness of generated SysML v2 models in 4 of 5 test scenarios.","lead":"SysTemp is a multi-agent system that uses several large language model agents, plus a template generator and a syntax parser, to turn natural language specifications into SysML v2 models. The paper reports a small ablation study suggesting the template step makes generated models syntactically valid more often than direct generation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'near-systematic convergence' claim rests on five single runs and a fixed five-iteration horizon, so the 4/5 versus 1/5 gap is not yet established.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: the empirical comparison is under-powered and uncontrolled. I agree with that assessment. I considered two alternative concerns: (1) the ParserAgent's Java validation tool is not released, so the dependent variable is not independently verifiable; and (2) the no-template baseline might converge if given more iterations. Both are real risks, but they are secondary to the sample-size problem and would mostly be resolved by a proper repeated-measures release with raw logs and an explicit iteration budget. The paper's appendices give enough prompt and template detail to re-implement the pipeline, which is a positive feature, but the headline conclusion that the template mechanism yields near-systematic syntactic convergence is supported by only five one-shot runs. The p-value for the reported 4/5 versus 1/5 contingency is around 0.10, so the observed gap is consistent with chance, and the fixed five-step horizon further inflates the apparent advantage. Given these issues, the reader's CONDITIONAL verdict is appropriate; no adjustment beyond the already requested revisions is needed.","tokens_in":8600,"tokens_out":7631,"duration_ms":85966,"concrete_test":"Run the ablation as a paired repeated-measures experiment: for each of the five scenarios, run K greater than or equal to 20 independent trials per condition (TG vs no-TG) on each of GPT-4 Turbo and Claude 3.5 Sonnet, using temperature greater than 0 and recorded seeds. Record convergence at step 5 and, separately, the first iteration at which the ParserAgent reports zero errors. Report per-model contingency tables, median/IQR of iterations-to-convergence, and a bootstrap 95% CI for the difference P(converge|TG) minus P(converge|no-TG). Also run the no-TG condition for 10 iterations to test the horizon effect. If the CI includes zero or no-TG converges by step 10 in most trials, the near-systematic claim should be withdrawn or softened to a suggestion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing evidence for the paper's central claim is Section 5's ablation: with TemplateGeneratorAgent, 4/5 scenarios converge to error-free syntax at step 5; without it, 1/5. This evidence is too weak to carry the 'near-systematic convergence' conclusion. Section 4.2 selects five bicycle-related use cases and two closed-source LLMs, and Section 3.1 explicitly models the LLM as a non-deterministic function f; yet no repeated trials, temperature settings, seeds, per-model/per-scenario counts, or variance measures are reported. Success is defined as error-free at a fixed fifth iteration, so a baseline that would converge on iteration 6 is counted as a failure, making the comparison horizon-limited. Under a Fisher exact test on the reported 4/5 vs 1/5 table, the difference is not significant (one-sided p approximately 0.10). The absence of released code or raw logs means the reported gap cannot be checked independently and could reflect seed or temperature luck rather than the template mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SysTemp, a multi-agent pipeline for generating SysML v2 concrete syntax from natural language specifications. The pipeline consists of four agents: a SpecificationGeneratorAgent that extracts a structured Python dictionary, a TemplateGeneratorAgent that produces a syntactic skeleton using Jinja2 rules, a WriterAgent that completes the skeleton, and a ParserAgent that validates the output against the OMG SysML v2 grammar and returns error feedback. The main empirical claim is an ablation: with the TemplateGeneratorAgent, 4 out of 5 scenarios converge to error-free syntax at step 5, whereas without it only 1 out of 5 converges, which the paper calls 'near-systematic convergence'. The evaluation uses five bicycle-related scenarios and two commercial LLMs, GPT-4 Turbo and Claude 3.5 Sonnet.","tokens_in":8814,"tokens_out":5458,"duration_ms":57118,"significance":"The problem is relevant and the proposed direction is sensible: SysML v2 is a low-resource formal language with a complex, evolving syntax, and decomposing generation into specification extraction, template-based skeleton creation, and iterative parser feedback is a plausible way to improve reliability. The formalization of the agents in Section 3.1 is clear, and the use of an external OMG grammar as the validation oracle is a strength. Appendix B gives concrete templating rules, which helps reproducibility of the skeleton generation. However, the paper's central empirical claim is not supported by the evidence as presented. Five single-run scenarios, with no repeated trials, no variance measures, no statistical test, and a fixed five-iteration horizon, are insufficient to establish 'near-systematic convergence'. The gap between 4/5 and 1/5 is not statistically significant under a Fisher exact test (one-sided p approximately 0.10). The contribution is therefore best viewed as a promising preliminary system description rather than a validated method.","major_comments":[{"comment":"The central claim 'near-systematic convergence' rests on the reported difference 4/5 with the TemplateGeneratorAgent versus 1/5 without it, at a fixed fifth iteration. Section 3.1 explicitly models the LLM as a non-deterministic function f, yet no repeated runs, temperature settings, seeds, or per-model/per-scenario counts are reported. Under the natural reading of the counts as a 2x2 table, Fisher's exact test gives a one-sided p-value of approximately 0.10, which does not establish the claimed improvement. In addition, success is defined as error-free at step 5, so a baseline that would converge at step 6 is counted as a failure, making the comparison horizon-limited. Please provide repeated trials with variance, a proper statistical test, and either relax the claim to a preliminary observation or substantiate it with more data. Releasing the code and raw logs would also allow independent verification of the reported trajectories.","section":"§5, Fig. 6"},{"comment":"The manuscript does not give a per-model or per-scenario breakdown of the results. Section 4.2 states that two LLMs, GPT-4 Turbo and Claude 3.5 Sonnet, were selected, and Section 5 says that 'both models yield similar results' and that GPT-4 has on average three fewer errors. However, it is unclear whether the 4/5 and 1/5 figures are pooled across the two models, whether each scenario was run once per model or once total, and what the error trajectories look like for each model and scenario. Please provide a full data matrix (scenario x model x condition x iteration) and state precisely which numbers correspond to which configuration.","section":"§4.2 and §5"},{"comment":"The ParserAgent serves both as the in-loop corrector and as the evaluation oracle. While the grammar is the external OMG SysML v2 standard and the parsing mechanism is implemented in Java, the parser tool P is not described in enough detail: it is not stated how syntax errors are counted (per token, per message, per location), nor whether the tool has been validated against a set of known-good and known-bad SysML v2 snippets. Since 'success' is defined as zero errors reported by P, a parser that under-reports errors would directly inflate the main result. Please characterize the parser's behavior and, if possible, report precision/recall on a small labeled set of SysML v2 fragments.","section":"§3.1.4 and §4.1"},{"comment":"The evaluation measures only syntactic correctness, while the abstract and introduction frame the contribution as improving the 'quality of the generations'. The TemplateGeneratorAgent produces the skeleton deterministically via Jinja2, and the WriterAgent prompt in Table 3 instructs the model never to change the template's structure. Under these conditions, the ablation partly measures the benefit of constraining the output format. Without any semantic metric, human review of the five final models, or at least a requirements-coverage check, it is unclear whether the generated models satisfy the natural-language specifications. The paper itself acknowledges in Section 6 that semantics are not addressed; please add a qualitative assessment or explicitly narrow the claims to syntactic convergence.","section":"§4.1 and §6"}],"minor_comments":[{"comment":"There are several typos and inconsistent terms: 'SysML v11' should likely be 'SysML v1', 'Engenieering' should be 'Engineering', and 'Pytthon' in Section 3 should be 'Python'.","section":"§1 and keywords"},{"comment":"Model naming is inconsistent: Section 4.2 says GPT-4 Turbo and Claude 3.5 Sonnet, Section 5 says GPT-4 and Claude Sonnet 3.5, and reference [24] is titled 'Introducing Claude 2.1'. Please align the model names and versions throughout.","section":"§4.2, §5, and reference [24]"},{"comment":"The formula defining Ex has mismatched braces: 'Ex = concat({(D(i) NL, Dict(i))| i∈{ 1, . . . , k})' is missing a closing brace. Please correct the notation.","section":"§3.1.1"},{"comment":"The caption should describe the axes, the unit of the error count, and whether the plotted lines are per-scenario, per-model, or average trajectories. Currently the reader cannot tell what is aggregated in the figure.","section":"Fig. 6 caption"},{"comment":"The prompts refer to a 'syntax_checker_agent' while the main text calls it ParserAgent; please unify the terminology. Also, the quotation marks around code fences in the prompt tables are inconsistent across Tables 2-5.","section":"Appendix A"},{"comment":"The sentence 'On average, three fewer errors but this difference is not substantial enough to warrant a significant distinction' is ambiguous: it should state which model has fewer errors, in which condition, and over how many scenarios this average is taken.","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The paper is best viewed as a short system-description / preliminary study. The central idea is interesting and the pipeline is clearly described, but the empirical basis is currently too thin for a journal-level claim of systematic improvement. Before acceptance, I would want to see repeated runs, a per-scenario data table, and a parser validation, or a substantially softened claim. Please also check reference [19], 'Multi-agent language models for low-resource language generation' (ICML 2021); it may be mis-cited or difficult to verify, and similar checks on other entries in the bibliography would be prudent. The fit with a cs.CL venue is acceptable if the empirical evaluation is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick read on arXiv:2506.21608. The new thing is narrow but real: the authors apply the familiar multi-agent code-generation recipe—specialized LLM agents plus parser feedback—to SysML v2 textual notation, and add a TemplateGeneratorAgent that produces a Jinja2-based skeleton so the LLM only fills in slots. That is a sensible way to tame a low-resource formal language, and the paper describes the architecture clearly enough to reimplement. The parser is the actual OMG Java parser, so the evaluation metric is objective, not the LLM grading its own output.\n\nThe problem is the evidence. Section 5 claims 'near-systematic convergence' on the strength of four out of five scenarios converging with the template agent and one out of five without. Those numbers come from a single run per scenario, two closed-source LLMs, no seeds, no temperature settings, and no repeated trials. A Fisher exact test on the 4/5 vs 1/5 table gives p around 0.10, so the gap is consistent with chance. The runs are also truncated at a fixed fifth iteration, so a baseline that would converge later is counted as a failure. No code or raw logs are released, so I can't check the counts. That is the load-bearing result, and it is too wobbly to carry the wording.\n\nThe milder concern is that both the correction loop and the evaluation use the same parser, which is slightly incestuous, but given that the grammar is an OMG standard this is more a limitation than a flaw. The bigger gap is semantic: the paper only measures syntax, and the conclusion acknowledges that the generated values haven't been checked for meaning. For MBSE practitioners, that leaves the practical value unproven.\n\nNone of this is fatal. The architecture is sensible, the paper is honest about being preliminary, and the authors explicitly list the missing pieces (benchmarks, semantic checks, variance). I would not desk reject this. It deserves a referee: it's a serious systems paper with a real parser-based metric, and the evaluation can be strengthened with more scenarios, repeated runs, and released code. I'd ask for those before acceptance.\n\nFor a reading group: it's a quick read if you work on LLM generation for formal languages, but it's a niche application, so maybe. I wouldn't cite it in my own work in the next year unless I moved into MBSE.\n\nRecommendation: engage with it in peer review, but push for a real evaluation before publication.","headline":"A worthwhile niche system for NL-to-SysML v2, but the 'near-systematic convergence' claim rests on five single runs and needs a real evaluation before it can be believed.","tokens_in":9311,"tokens_out":3182,"would_cite":false,"duration_ms":33933,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Inserting a rule-based template generator and a parser-feedback loop makes LLM-produced SysML v2 models syntactically valid in 4 of 5 tested scenarios, versus 1 of 5 without the template.","keywords":["SysML v2","multi-agent system","template-based generation","large language models","model-based systems engineering","syntax validation","few-shot prompting"],"falsifier":"Run each of the five scenarios many times (for example, 20 repeated draws) with and without the TemplateGeneratorAgent, keeping the same iteration budget and parser, and check whether the with-template success rate stays clearly above the without-template success rate across repetitions; the claim collapses if the two conditions converge at comparable rates once variation is measured.","tokens_in":8433,"feed_emoji":"⚙️","tokens_out":6596,"duration_ms":63685,"temperature":0.7,"pith_summary":"Auto-generating SysML v2 models from natural language is hard because the language is young, sparsely documented, and has a strict syntax. The paper proposes SysTemp, a multi-agent pipeline whose key addition is a TemplateGeneratorAgent that turns an LLM-extracted structured specification into a rule-based SysML v2 skeleton before a writer agent fills in details. A separate parser agent checks each draft and feeds error reports back to the writer. In an ablation on five bicycle-related scenarios, the template route produced parser-validated syntax in 4 of 5 cases, versus 1 of 5 without the template, with similar results for two closed-source LLMs. The finding offers a concrete recipe for making a low-resource formal language tractable: constrain the LLM with a validated syntactic scaffold and then repair iteratively.","feed_headline":"Template skeleton lifts SysML v2 generation success from 20% to 80%","feed_subtitle":"Adding a rule-based skeleton and a parser-feedback loop lets LLMs produce valid SysML v2 models in most test cases.","key_machinery":"The TemplateGeneratorAgent is the component that carries the argument. It is an LLM-augmented tool: the LLM converts the extracted dictionary into the tool's expected signature, and a rule-based template then substitutes package, requirement, attribute, and constraint values into a fixed SysML v2 skeleton. Because the skeleton is produced by deterministic rules rather than by the LLM, the enclosing syntax is guaranteed before the writer begins; the LLM only fills in content. The ParserAgent then validates each completed draft against the formal grammar and returns location-tagged errors, which the WriterAgent uses to revise, forming a closed repair loop.","core_discovery":"The paper's central claim is that a rule-based template agent is the difference between failure and near-systematic success in LLM-generated SysML v2. Concretely, on five bicycle-related natural-language specifications, the full SysTemp pipeline (specification extraction into a structured dictionary, template-skeleton generation, writer completion, and parser feedback) ended with a parser-validated, syntactically correct model in 4 of 5 scenarios. Without the template agent, the same write-and-repair loop converged in only 1 of 5 scenarios. The two evaluated closed-source LLMs behaved similarly, with one showing a small average reduction in error count (about three fewer errors), which the paper does not treat as significant. The conclusion concerns syntax only: the parser certifies that the generated text conforms to the language grammar, not that the semantic content is correct.","pith_inferences":["If the template effect holds at scale, the practical lesson for low-resource formal languages is to invert the usual prompting advice: rather than asking the model to produce the whole artifact, provide a grammar-derived scaffold and reduce the generative task to content filling.","The syntax-only success metric leaves open how much of the generated model is semantically faithful to the requirements; an immediate follow-up would be an automated check that each stated requirement appears as a requirement block with the intended property values.","Because the parser is the only piece that depends on SysML v2 specifically, the same orchestration should transfer to other formal languages simply by swapping the parser and the template rules; a cheap test would be to apply SysTemp unchanged to a second language with a small grammar."],"forward_implications":["Generating a rule-based skeleton before asking the LLM to fill in details is enough to reach parser-valid syntax in 4 of 5 tested scenarios; without the skeleton the success rate drops to 1 of 5.","The benefit is not tied to a particular provider: both tested closed-source LLMs reach the same convergence rate, so the mechanism transfers across models.","The write-and-repair loop converges within the observed iteration budget when the skeleton is present, which means the ParserAgent's error reports are actionable enough for the writer to correct them.","The pipeline's explicit future applications include semantic refinement of generated values and synthetic data generation for this low-resource language, both of which become more feasible once syntax is reliably correct."],"supporting_citations":[{"why":"Supplies the few-shot prompting technique that lets the SpecificationGeneratorAgent extract a structured dictionary from natural language.","marker":"[8]"},{"why":"Provides the multi-agent conversation-orchestration pattern that the four-agent SysTemp pipeline adapts.","marker":"[20]"},{"why":"The prior method the paper contrasts: it requires an existing model as input, while SysTemp generates from scratch.","marker":"[18]"},{"why":"One of the two closed-source LLMs evaluated in the ablation study; the results are reported for it.","marker":"[12]"},{"why":"The other closed-source LLM evaluated in the ablation; the paper reports similar behavior to the first.","marker":"[24]"},{"why":"Surveys LLM multi-agent systems and supports the claim that splitting a complex task into specialist subtasks helps.","marker":"[16]"},{"why":"Supports the design choice to generate the shorter concrete syntax rather than the abstract JSON syntax, to avoid long-context degradation.","marker":"[17]"}],"fun_headline_variants":["Template agent boosts valid SysML v2 from 1 in 5 to 4 in 5","Rule-based skeleton lifts SysML v2 syntax success to 80%","Multi-agent template system quadruples valid SysML v2 output","Parser feedback plus template skeleton hits 80% SysML v2 validity","SysTemp: template skeleton raises SysML v2 generation from 20% to 80%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that five single runs on the selected scenarios, without repeated trials or variance measures, are enough to attribute the 4-out-of-5 versus 1-out-of-5 gap to the template mechanism rather than to random variation or prompt differences.","fun_headline_variants_meta":{"raw":{"variants":["Template agent boosts valid SysML v2 from 1 in 5 to 4 in 5","Rule-based skeleton lifts SysML v2 syntax success to 80%","Multi-agent template system quadruples valid SysML v2 output","Parser feedback plus template skeleton hits 80% SysML v2 validity","SysTemp: template skeleton raises SysML v2 generation from 20% to 80%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1284,"prompt_tokens":807,"completion_tokens":477,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":423,"tokens_out":477,"duration_ms":4020,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:14:18.464049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each of the five scenarios many times (for example, 20 repeated draws) with and without the TemplateGeneratorAgent, keeping the same iteration budget and parser, and check whether the with-template success rate stays clearly above the without-template success rate across repetitions; the claim collapses if the two conditions converge at comparable rates once variation is measured.","supporting_citations":[{"cited_title":"Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared Kaplan et al","cited_arxiv_id":null,"evidence_quote":"Supplies the few-shot prompting technique that lets the SpecificationGeneratorAgent extract a structured dictionary from natural language."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior method the paper contrasts: it requires an existing model as input, while SysTemp generates from scratch."},{"cited_title":"Package\" :","cited_arxiv_id":null,"evidence_quote":"The other closed-source LLM evaluated in the ablation; the paper reports similar behavior to the first."}],"review_version":1}