{"id":"07552ea8-b6b0-47f8-88ea-b16366cb9c96","arxiv_id":"2501.07553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BERTiMuS generates Simulink model mutants via CodeBERT masked-token prediction, and in a requirements-aware evaluation it outperforms the FIM tool.","lead":"This paper introduces BERTiMuS, a tool that uses the CodeBERT language model to automatically create mutated versions of Simulink models for testing control systems. It reports that when mutants are judged against system requirements, BERTiMuS outperforms the existing FIM tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Requirements-aware comparison hinges on unverified requirement oracles and reference suites; the 128-vs-61 gap may be an artifact of sparse requirements (ATCS:1, AECS:2) rather than mutant quality.","rationale":"The reader's weakest_assumption correctly identifies the reference test suites and requirement sets as the load-bearing condition. My analysis sharpens this: the paper does not even describe the requirements oracle, and the extreme imbalance in requirement counts (ATCS: 1, AECS: 2) makes the requirements-aware comparison particularly fragile. The internal table arithmetic is not contradictory if the 'could not be killed' row is read as 'killable mutants not killed by the other approach's selected tests,' so the main threat is the unverified, possibly sparse, and inconsistently demanding requirement sets and the adequacy of adaptive-random test suites for triggering requirement violations. This is a correctness-risk concern, not a consensus disagreement. The paper's tool design, fine-tuning procedure, and RQ1 analysis are credible; the weakness is specifically in the empirical support for Finding 3. The reader already returned a CONDITIONAL verdict, which appropriately reflects that the central claim is plausible but not yet robustly supported. I therefore see no need to change the verdict; the requested conditional revisions (statistical analysis, complete artifact, and detailed test-suite/requirement specification) should include the requirement oracle and killability recomputation I propose.","tokens_in":8740,"tokens_out":7507,"duration_ms":66606,"concrete_test":"Request the authors to: (1) publish the exact requirement specifications (e.g., STL properties or Simulink assertion blocks) for all five models; (2) for each model and each tool, compute the number of mutants that are killed by the full reference suite under the requirements-aware notion, i.e., the true 'killable' set; and (3) recompute Table III using only mutants that are requirements-aware killable by the full reference suite. If the killable counts drop below 311/402 or the 128-vs-61 gap changes materially, the finding is not robust. Additionally, independently derive requirement properties for the Lockheed Martin models from the cited benchmark [23] and for ATCS/AECS from any available sources, then re-run the comparison with these formalized requirements to check whether the gap persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Finding 3) is that BERTiMuS outperforms FIM under requirements-aware mutation testing, supported by Table III: test cases selected by FIM fail to kill 128 BERTiMuS mutants, while test cases selected by BERTiMuS fail to kill only 61 FIM mutants. This comparison is meaningful only if (a) the requirement sets for all five subject models are complete, precise, and equally demanding, and (b) the reference test suites, generated by adaptive random testing, are adequate to expose requirement violations. The paper provides neither. It never states how requirements are formalized (e.g., temporal logic properties, Simulink assertion blocks) or how a requirement violation is detected during simulation. For ATCS and AECS (the models from the FIM baseline), only 1 and 2 requirements are listed; BERTiMuS may by chance perturb blocks tied to those sparse requirements, while FIM's 402 mutants touch many blocks unrelated to them, making the comparison an artifact of the requirement set rather than of mutant representativeness. Additionally, if the adaptive-random reference suite does not contain test cases that drive the model into states where the (few) requirements are violated, then genuinely requirements-relevant mutants are counted as 'not killed by the other approach' simply because no test in the suite can kill them. The 128-vs-61 gap could shrink or reverse once these confounds are controlled. The table's mutation scores (77% vs 78%) are nearly identical, so the headline result rests entirely on the unbalanced 'not killed' counts, which are highly sensitive to the requirement and test-suite choices.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BERTiMuS, a mutation testing approach for Simulink models that converts models into JSON textual representations, masks block-property tokens, and replaces them with the top-k predictions of a CodeBERT model fine-tuned on Simulink models from SLNET. The approach is evaluated on five industrial models from two benchmarks, comparing with the FIM tool. The paper reports that BERTiMuS covers all documented block-based Simulink mutation patterns, that BERTiMuS and FIM are complementary in terms of selected test cases, and that under a requirements-aware notion of mutation testing, BERTiMuS outperforms FIM (Table III).","tokens_in":9032,"tokens_out":8012,"duration_ms":70580,"significance":"The paper addresses an interesting and timely problem: applying masked-language-model-based mutation generation to Simulink models. Its strengths include the use of an external baseline tool (FIM) for comparison, a fine-tuning corpus (SLNET) that is independent of the subject models, a replication package, and a clear statement of novel claims (complementarity and requirements-aware superiority). If the central claims hold after addressing the measurement issues, the work would be a useful contribution to mutation testing for model-based development.","major_comments":[{"comment":"In Table III, the row 'Average # of killed mutants that could not be killed by the test cases selected by the other approach' reports 128 for BERTiMuS and 61 for FIM under requirements-aware mutation testing. These raw counts are used in Finding 3 and the abstract to claim that 'test cases selected using FIM mutants failed to kill twice as many BERTiMuS mutants as the number of FIM mutants not killed by test cases selected using BERTiMuS mutants.' However, the denominators differ: BERTiMuS has 311 killable mutants and FIM has 402. The paper should report these counts as proportions (e.g., 128/311 ≈ 41% vs 61/402 ≈ 15%) and discuss whether the absolute-count framing is appropriate. As written, the claim conflates the size of the mutant sets with the difficulty of killing them.","section":"Section IV, Table III and Finding 3"},{"comment":"The paper defines requirements-aware mutation testing as killing a mutant when it violates a requirement satisfied by the original model, citing Bartocci et al. [4]. However, it never states how the requirements for the five models (Table I) are represented or checked. No property language, assertion mechanism, or monitor tool is described. This is a load-bearing omission because Finding 3 depends entirely on a reliable oracle for requirement violations. Without this specification, the results are not reproducible, and the sparse requirement counts for ATCS (1) and AECS (2) raise the possibility that the comparison is an artifact of the particular requirement sets rather than of mutant quality.","section":"Section IV, RQ2 experiments"},{"comment":"The sentence 'For both FIM and BERTiMuS, we select a minimal subset T ⊆ TS of test cases that can kill as many mutants generated by each approach as possible' is the entire description of the test-set selection procedure. The paper does not specify how minimality is achieved (e.g., greedy, optimal, or heuristic), how ties are broken, or what randomness is involved in the five repetitions. This makes the complementarity findings (Figure 3) impossible to reproduce and weakens the comparison between the selected test cases of the two approaches.","section":"Section IV, RQ2 experiments"},{"comment":"The reference test suites are generated by 'adaptive random testing, as used in earlier studies,' but no details are given about the size of the suites, the input value ranges, or the stopping criteria. The adequacy of these suites directly determines which mutants are considered 'killable' (Table III reports only 311/387 BERTiMuS and 402/594 FIM mutants as killable). The paper should provide the suite sizes and a justification that the suites are sufficiently powerful to expose requirements violations, particularly for the models with very few requirements.","section":"Section IV, RQ2 experiments"},{"comment":"Table III reports only aggregate averages across the five models, without per-model breakdown. Given the heterogeneity of the models (57 to 825 blocks; 1 to 32 requirements; and two models from the baseline's own benchmark), the authors should show per-model results to demonstrate that Finding 3 is not driven by a single model or by the models with extremely sparse requirement sets.","section":"Table III and Table I"}],"minor_comments":[{"comment":"There appears to be a duplicate/overlapping figure: the text under 'Figure 2' contains two sets of (a)-(d) subfigures, with a caption appearing in the middle of the description. This should be cleaned up so that the figure and caption show a single coherent illustration.","section":"Section II, Figure 2"},{"comment":"The analysis claiming that all BERTiMuS mutants can be generated using the patterns in Table II is not described. The paper should specify the classification procedure and provide a mapping of mutants to patterns to substantiate the coverage claim.","section":"Section IV, RQ1"},{"comment":"The statement that BERTiMuS generates 'mutant instances that are unlikely to be generated based on syntactic and manual mutation rules' seems to conflict with the earlier claim that all mutants can be generated using the 10 patterns in Table II; this tension should be clarified.","section":"Section IV, RQ1"},{"comment":"The hyperparameter choices (top_k=3, fine-tuning for 15 epochs, masking 15%) are given without sensitivity analysis; the paper should at least acknowledge the influence of these choices on the reported results.","section":"Section III and IV"},{"comment":"The table reports averages as integers because 'there were almost no differences across the five runs'; the paper should provide the actual spread (e.g., min/max or standard deviation) to support this statement.","section":"Section IV, Table III"},{"comment":"The paper lacks a threats-to-validity section, which would be important given the small number of subjects and the manual analysis in RQ1.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is potentially a good fit for the journal, but the authors must address the normalization of mutant counts, the missing requirement oracle specification, and the underspecified test-subset selection. The use of only five models and the absence of per-model results are also concerns. I would like to see per-model tables and the exact selection algorithm before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a legitimate adaptation of the MLM mutation idea to Simulink, with a clean pipeline and an honest comparison to FIM. But the headline result—that under requirements-aware killing BERTiMuS beats FIM—rests on an evaluation that I don't trust yet. The stress-test note is right: with ATCS having one requirement and AECS two, the 128-vs-61 gap could be a quirk of the requirement set rather than mutant quality.\n\nWhat's good: the masking-and-predicting approach is clearly described, the fine-tuning corpus (SLNET) is independent of the subject models, and RQ1's confirmation that all generated mutants fall within known block patterns is a useful sanity check. The complementarity finding (Finding 1) is plausible and supported by the pie charts. Using a reference suite generated by adaptive random testing and repeating runs five times is a reasonable start.\n\nSoft spots: the requirements-aware comparison has a load-bearing weakness. The paper doesn't say how the requirements are formalized or how a violation is detected during simulation. With so few requirements for two of the five models, a mutant that touches those blocks will be counted as 'not killed' if the reference suite doesn't drive the model into the violating state. The mutation scores are nearly identical (77% vs 78%), so the whole argument rests on the unbalanced 'not killed' counts, which are sensitive to test suite adequacy. No significance tests or confidence intervals. The minimal test subset selection procedure is only sketched—I can't tell exactly how it handles ties or whether the five runs are truly independent. And the replication package is an anonymous bit.ly link; there's no commit hash to pin down what was run.\n\nWhere does that leave us? The paper is a proof-of-concept that CodeBERT can generate plausible Simulink mutants, and that's worth something to the simulation-testing community. But the requirements-aware superiority claim is not yet established. I'd send it to a serious referee—the topic is niche but real—and expect major revisions on the evaluation. If the authors add per-model results, proper requirement oracles, and statistical tests, the paper could become solid.\n\nRecommendation: engage with it, but treat Finding 3 as a hypothesis, not a result.","headline":"Genuine adaptation of MLM mutation to Simulink, but the requirements-aware superiority claim is undercut by an unverified oracle and sparse requirement sets.","tokens_in":9611,"tokens_out":2791,"would_cite":false,"duration_ms":27200,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A CodeBERT-based mask-and-predict pipeline generates Simulink mutants that, under requirements-aware mutation testing, outperform a state-of-the-art tool.","keywords":["Simulink mutation testing","CodeBERT","masked language modeling","requirements-aware mutation testing","mutant generation","fault injection","LLM-based testing","model-based testing"],"falsifier":"A replication study on a broader set of Simulink models with independently validated requirement specifications and reference test suites constructed by a different method (for example, search-based generation) could falsify the claim: if, under requirements-aware mutation testing, the number of FIM mutants left unkilled by BERTiMuS-selected tests equals or exceeds the number of BERTiMuS mutants left unkilled by FIM-selected tests, the claimed outperformance would not hold up.","tokens_in":8524,"feed_emoji":"🧪","tokens_out":9258,"duration_ms":70513,"temperature":0.7,"pith_summary":"BERTiMuS generates mutants for Simulink models by converting the model to a simplified JSON text, masking individual block-property tokens, and replacing them with predictions from CodeBERT, a transformer model fine-tuned on 2,611 third-party Simulink models. The paper claims that this single mechanism covers every block-based Simulink mutation pattern catalogued in the literature, producing mutants such as swapped GotoTag labels and renamed Stateflow variables that rule-based operators would need bespoke logic to generate. Compared against FIM, a state-of-the-art Simulink mutation tool, the two approaches select complementary test sets; under the classical output-based notion of killing they are fairly similar, with FIM slightly ahead in mutant count and difficulty. Under the requirements-aware notion, where a mutant is killed only by a test that exposes a violation of a system requirement, BERTiMuS is claimed to outperform FIM: tests selected using FIM mutants fail to kill 128 BERTiMuS mutants, while tests selected using BERTiMuS mutants fail to kill only 61 FIM mutants. If correct, this indicates that a general-purpose language model, without hand-coded fault patterns, can generate mutants that are more representative of requirements-relevant faults and harder to detect.","feed_headline":"CodeBERT mutants stump FIM-selected tests twice as often","feed_subtitle":"The CodeBERT pipeline creates requirement-relevant faults that FIM-selected tests miss","key_machinery":"The load-bearing mechanism is the mask-and-predict pipeline. BERTiMuS converts a Simulink model's XML serialization into a simplified JSON text, masks a block property token, and asks CodeBERT to predict the masked token; the top k predictions that differ from the original value and produce compilable models become mutants. CodeBERT is fine-tuned on a corpus of 2,611 third-party Simulink models so that its predictions respect Simulink structure and naming conventions. This context-aware prediction is what lets BERTiMuS generate mutants like swapping a GotoTag label between safety-level signals or renaming a Stateflow variable to another variable of the same type already in the model.","core_discovery":"The central claim is that a mask-and-predict strategy with CodeBERT is a viable and, under one important criterion, superior way to generate mutants for graphical Simulink models. BERTiMuS first serializes the model into a JSON representation, masks each block property, and lets a fine-tuned CodeBERT propose alternatives; the top k compilable predictions become mutants. The paper shows these mutants instantiate all ten block-based mutation pattern families reported in the Simulink testing literature. When compared with FIM on five industrial-benchmark models, BERTiMuS selects a smaller but partially non-overlapping set of test cases, and under requirements-aware mutation killing the asymmetry in Table III is the paper's headline result: the test cases selected via FIM mutants fail to kill 128 BERTiMuS mutants, whereas the test cases selected via BERTiMuS mutants fail to kill only 61 FIM mutants. The authors interpret this as evidence that BERTiMuS mutants represent requirements-relevant faults that are harder to detect than FIM mutants under the same test-selection procedure.","pith_inferences":["Beyond the paper: if the requirements-aware advantage generalizes beyond the five studied models, LLM-based mutant generation could serve as a low-cost requirements oracle amplifier, suggesting which model behaviors deserve requirement-level scrutiny without an explicit fault model.","Beyond the paper: the approach's quality is likely bounded by the diversity of the fine-tuning corpus of Simulink models; a larger or more domain-specific corpus might produce even more requirement-relevant mutants, a testable prediction.","Beyond the paper: the mask-and-predict mechanism might transfer to other graphical modeling notations with XML serializations, although the paper does not evaluate that extension."],"forward_implications":["BERTiMuS generates mutants that cover all ten block-based Simulink mutation pattern families catalogued in the literature, so it could substitute for hand-coded pattern libraries at the block level.","BERTiMuS and FIM select different test cases and kill different mutants, making the two approaches complementary rather than one subsuming the other.","Under classical mutation testing, FIM produces more and harder-to-kill mutants; under requirements-aware mutation testing, the asymmetry reverses, with BERTiMuS mutants being twice as hard to kill by FIM-selected tests.","Requirements-aware mutation testing is a stricter and more meaningful criterion than classical output-based killing, and it changes the rank order of the two tools."],"supporting_citations":[{"why":"the pre-trained masked language model whose predictions form the generated mutants","marker":"[10]"},{"why":"the approach from which the mask-and-predict strategy and the comparison procedure are adapted","marker":"[8]"},{"why":"the state-of-the-art baseline tool and the source of two subject models","marker":"[3]"},{"why":"the notion of requirements-aware mutant killing used in the evaluation","marker":"[4]"},{"why":"the corpus of third-party Simulink models used for fine-tuning CodeBERT","marker":"[28]"},{"why":"the source of several block-based mutation patterns and the adaptive random testing procedure for reference suites","marker":"[21]"},{"why":"the mutation testing survey from which block-based pattern families in Table II are drawn","marker":"[25]"}],"fun_headline_variants":["CodeBERT mutants stump FIM tests twice as often","BERTiMuS: CodeBERT-driven mutants beat FIM in requirements testing","Mask-and-predict mutants outsmart FIM on Simulink","CodeBERT generates harder-to-kill Simulink mutants than FIM","For requirement-aware mutation, CodeBERT outperforms FIM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the assumption that the five reference test suites (built by adaptive random testing) and the listed requirement sets for the five models give a fair, equally demanding yardstick for both BERTiMuS and FIM; if the requirements are incomplete or the test suites are biased, the 128-versus-61 asymmetry may be an artifact of the benchmark rather than of the approach.","fun_headline_variants_meta":{"raw":{"variants":["CodeBERT mutants stump FIM tests twice as often","BERTiMuS: CodeBERT-driven mutants beat FIM in requirements testing","Mask-and-predict mutants outsmart FIM on Simulink","CodeBERT generates harder-to-kill Simulink mutants than FIM","For requirement-aware mutation, CodeBERT outperforms FIM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3370,"prompt_tokens":910,"completion_tokens":2460,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":2371}},"tokens_in":526,"tokens_out":2460,"duration_ms":18064,"temperature":1.0,"reasoning_tokens":2371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:37:56.142594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication study on a broader set of Simulink models with independently validated requirement specifications and reference test suites constructed by a different method (for example, search-based generation) could falsify the claim: if, under requirements-aware mutation testing, the number of FIM mutants left unkilled by BERTiMuS-selected tests equals or exceeds the number of BERTiMuS mutants left unkilled by FIM-selected tests, the claimed outperformance would not hold up.","supporting_citations":[{"cited_title":"µbert: Mutation testing using pre-trained language models","cited_arxiv_id":null,"evidence_quote":"the approach from which the mask-and-predict strategy and the comparison procedure are adapted"},{"cited_title":"Fim: fault injection and mutation for simulink","cited_arxiv_id":null,"evidence_quote":"the state-of-the-art baseline tool and the source of two subject models"},{"cited_title":"Property-based mutation testing","cited_arxiv_id":null,"evidence_quote":"the notion of requirements-aware mutant killing used in the evaluation"},{"cited_title":"SLNET: A redistributable corpus of 3rd-party simulink models","cited_arxiv_id":null,"evidence_quote":"the corpus of third-party Simulink models used for fine-tuning CodeBERT"},{"cited_title":"Test generation and test prioritization for simulink models with dynamic behavior","cited_arxiv_id":null,"evidence_quote":"the source of several block-based mutation patterns and the adaptive random testing procedure for reference suites"},{"cited_title":"Mutation testing advances: an analysis and survey","cited_arxiv_id":null,"evidence_quote":"the mutation testing survey from which block-based pattern families in Table II are drawn"}],"review_version":1}