{"id":"ff02c1f6-284c-41ac-86ca-a37dd1a5e96c","arxiv_id":"2504.18827","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MMT4NL is a metamorphic-testing framework that reveals how often LLMs give inconsistent answers in in-context learning when inputs are perturbed, and it shows that GPT-4o and Gemini-2.0-Flash often fail these consistency checks.","lead":"This paper proposes MMT4NL, a testing framework that applies metamorphic testing to check whether large language models stay consistent when their inputs are slightly changed, such as replacing words with synonyms or adding typos. The authors tried it on GPT-4o and Gemini-2.0-Flash for sentiment analysis and question answering, finding many cases where the same question gets different answers under small changes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Metamorphic relations in Section 4 assume invariance under perturbations that are not always meaning-preserving; reported pass-rate drops may reflect legitimate meaning changes, not LLM bugs.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and the weakest assumption identified—validity of the metamorphic relations astest oracles—is precisely the most load-bearing concern. The paper asserts invariance relations in Section 4 without operationalizing 'meaning-preserving' or validating that each perturbation type satisfies the relation on the actual data. The paper's own qualitative examples in Section 7.3 show cases where the perturbation plausibly changes meaning, yet the paper labels the model's changed output as a bug. If these MRs are not valid, then the quantitative pass rates in Tables 3–5 do not measure trustworthiness; they measure sensitivity to input changes that may legitimately alter the answer. The framework's novelty as a software-testing-inspired evaluation concept remains, but its empirical claims and practical utility hinge on MR validity. A human-annotation check directly tests this assumption and would settle whether the reported failures are genuine bugs or artifacts of over-strong oracles. This does not change the reader's conditional verdict; it sharpens the condition that must be met for acceptance.","tokens_in":11051,"tokens_out":4959,"duration_ms":52752,"concrete_test":"Recalculate pass rates using only transformations that human annotators confirm preserve the expected answer. Sample at least 50 transformed inputs across all MR types (e.g., 10 per MR from the actual test sets). Have three annotators independently provide the expected sentiment or yes/no answer for both the original and the transformed input, and compute inter-annotator agreement. If annotators judge that the expected answer changes for more than 10% of the sampled transformations (i.e., the MR's invariance assumption fails), then the corresponding MRs are invalid and the reported pass rates overstate model inconsistency. Re-run the MMT4NL comparison on the subset of transformations that humans confirm are meaning-preserving; if the corrected pass rates are near 100%, the central claim of revealed bugs collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that MMT4NL reveals LLM interpretation flaws—depends on the validity of the metamorphic relations defined in Section 4. Several of these relations assert invariance that the paper's own examples contradict. For Taxonomy, the MR is P(x)=P(f(x)) for synonym substitution, but the paper reports (Section 7.3) that GPT-4o classified 'I'm really hungry' as Neutral yet 'I'm really starving' as Negative, and 'This software is very complicated' as Negative but 'very complex' as Neutral. These are not obviously bugs: 'starving' and 'complex' carry different intensity or connotation than 'hungry' and 'complicated', so a rational model could legitimately change its sentiment label. The same issue affects Robustness: the MR P(x)=P(f(x)) for typos is applied to 'tired' → 'tried', which is not a minor spelling error but a different word with different meaning; the model's changed output may be correct. Negation Handling is formally mis-specified as ¬P(x)≈P(f(x)) for a three-class sentiment task, and the QA example 'not uncommon' does not clearly invert the original question's meaning. For Fairness, changing 'Chief Justice' to 'female Chief Justice' alters the set of entities being asked about, so a changed answer can be factually correct rather than biased. Because these MRs are asserted rather than validated, the low pass rates reported in Tables 3–5 may be false positives: the model is not violating a trustworthy-invariance requirement; it is responding to a genuinely different input. Without evidence that the transformations preserve the ground-truth answer, the framework's headline finding—that it 'effectively revealed LLM interpretation flaws'—is not supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MMT4NL, a metamorphic-testing framework for evaluating in-context learning in LLMs. It defines nine perturbation categories (Taxonomy, NER, Negation, Vocab, Fairness, Robustness, Temporal, SRL, Coreference) as metamorphic relations and introduces pass rate as a summary metric. The authors apply the framework to GPT-4o and Gemini-2.0-Flash on a 50-item sentiment dataset and a 50-item StrategyQA subset under zero-, one-, and few-shot prompting, with and without context for QA. They report pass rates in Tables 3–5 and manually inspect selected failures to argue that MMT4NL reveals interpretation flaws and biases in state-of-the-art LLMs.","tokens_in":11334,"tokens_out":5254,"duration_ms":51936,"significance":"The central observation that LLM outputs are unstable under small linguistic perturbations is empirically demonstrated and useful for the SE-for-AI community. The paper's strengths are its explicit use of metamorphic relations as external oracles, the absence of fitted parameters or result-dependent calibration, and the reproducible experimental structure (scripts, prompts, and data via the public repository). If the metamorphic relations were validated as meaning-preserving, the framework would provide a practical complement to accuracy-based evaluation. However, the current evidence is weakened by the oracle-validity and analysis issues discussed below, so the significance is conditional on a substantial revision.","major_comments":[{"comment":"Several metamorphic relations assert output invariance under perturbations that the paper's own examples show are not meaning-preserving. The Taxonomy examples in Section 7.3, \"I'm really hungry\" → \"I'm really starving\" and \"very complicated\" → \"very complex\", change intensity and connotation, so a sentiment change can be a legitimate semantic response rather than a bug. The Robustness example in Table 2, \"tired\" → \"tried\", is not a minor typo but a different word, and the Fairness example in Section 7.3, \"Chief Justice\" → \"female Chief Justice\", changes the referent, so a changed answer can be factually correct. Because these metamorphic relations are asserted rather than validated (for example, with human annotation), the low pass rates in Tables 3–5 may substantially overstate the number of genuine LLM flaws. The authors should either restrict MRs to demonstrably meaning-preserving transformations, validate the transformations empirically, or reframe the reported rates as sensitivity rather than bugs.","section":"Section 4, Table 2, Section 7.3"},{"comment":"The Negation Handling MR is stated as ¬P(x) ≈ P(f(x)) for adding negation cues, but the examples do not implement a well-defined negation operation. For QA, \"would it be not uncommon to find a penguin in Miami?\" is not the negation of \"would it be common to find a penguin in Miami?\" — it introduces a double negative whose yes/no answer is not simply inverted. For sentiment, \"I'm so not energetic\" is not a negation of \"I'm so tired.\" Consequently, the Negation pass rates in Tables 3–5 do not measure what the section claims, and the low rates cannot be attributed to a specific negation-handling failure.","section":"Section 4, Negation Handling"},{"comment":"The experimental basis is too thin for the quantitative claims. Each task uses only 50 records, perturbations are mostly manually constructed, and results are reported as point-estimate pass rates without confidence intervals or significance tests. For instance, a difference between 84% and 88% in Table 3 corresponds to two examples out of 50, and the claim in Section 7.1 that MMT4NL \"effectively revealed LLM interpretation flaws\" rests on such small counts. The paper should provide uncertainty quantification, statistical comparisons across conditions, and a clear statement of how perturbations were generated and checked.","section":"Section 5, Section 6.2, Tables 3–5"},{"comment":"The manual analysis is presented as evidence of specific bug categories, but the authors state they \"only showed surprising and severe scenarios\" and report subset-relative rates such as 2/7, 3/8, and 5/7 without defining the denominator. This selective reporting cannot support general claims such as \"Action Verb Replacement caused the highest failure rates\" or \"models anchor sentiment inconsistently.\" The authors should report the full failure inventory, define the subsets, and provide a systematic error analysis rather than selected examples.","section":"Section 7.3"}],"minor_comments":[{"comment":"The conclusion twice uses \"NNT4ML\" instead of \"MMT4NL\" (\"We presented NNT4ML\" and \"NNT4ML can be applied\").","section":"Section 8"},{"comment":"The table entry \"SLR QA\" should be \"SRL QA\" to match the terminology used in Sections 3 and 4.","section":"Table 2"},{"comment":"The formula for pass rate has inconsistent spacing and subscripts (\"Ntest,N pass\") and should be typeset properly as N_pass / N_test.","section":"Definition 2"},{"comment":"Only the one-shot prompt templates are shown in the paper; the zero-shot and few-shot templates are not specified, so the experimental design is not fully reproducible from the text alone. The repository is referenced, but the paper should describe all templates or include them in an appendix.","section":"Section 6.2"},{"comment":"The dataset provenance is thin: the Chat Sentiment dataset is cited only as a Kaggle dataset without version or license information, and StrategyQA is referenced via a Hugging Face user page rather than the original dataset publication.","section":"Section 5"},{"comment":"The failure rates reported in the bullet points (e.g., 2/7 and 5/7) are not tied to the 50-item test sets described in Section 5; the paper should define the subsets from which these denominators come.","section":"Section 7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to interest the SE-for-AI community, but the current framing overstates the findings. The main revision should focus on validating or re-scoping the metamorphic relations and on a more rigorous statistical and error-analysis treatment. There are no citation-ethics concerns; the self-citations are to prior relevant work and are not used to force the results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Christian, quick take on arXiv:2504.18827. This is a methodology paper that recombines CheckList's metamorphic-testing vocabulary with PromptBench's adversarial perturbation ideas and applies the result to in-context learning. The empirical core is credible: GPT-4o and Gemini-2.0-Flash do flip answers on small perturbations, and the paper's tables give concrete pass rates across sentiment, QA with/without context, and zero/one/few-shot. That makes it a useful data point for anyone building LLM evaluations.\n\nWhat's genuinely new: a named framework (MMT4NL) with formal metamorphic relations, a pass-rate metric for ICL, and a systematic sweep across prompt regimes. The observation that few-shot prompting does not reliably improve robustness is worth recording. The writing is clear, and the artifact link is a plus. Also, the self-citations are to the authors' own prior NLP work; nothing circular there.\n\nWhere it gets soft: several metamorphic relations are not actually meaning-preserving, so the pass rates overstate the model's instability. Taxonomy requires P(x)=P(f(x)) for synonyms, but 'hungry' vs. 'starving' and 'complicated' vs. 'complex' are not interchangeable in sentiment. Robustness uses 'tired' → 'tried' as if it were a typo, but that is a different word with a different meaning. Negation's formal MR ¬P(x)≈P(f(x)) is mis-specified for three-class sentiment, and the QA example 'not uncommon' does not truly invert the question. Fairness's 'female Chief Justice' changes the referent, so a changed answer may be correct. These are not minor nits; they undermine the quantitative claim that MMT4NL 'effectively revealed' bugs. The paper itself even shows failure examples that are reasonable model behavior, which complicates the interpretation.\n\nOther concerns: 50 records per task is small, there are no error bars or repeated runs, and the perturbations were manually constructed except for typos. The selective display of 'surprising' examples is fine for illustration but not for evidence. If the authors fix the MR definitions, add uncertainty bounds, and soften the conclusions, the framework could be useful.\n\nThis paper deserves a serious referee: the question of whether metamorphic testing is a valid oracle for LLM behavior is worth airing. I would not desk-reject it, but I would send it for review expecting major revision.","headline":"A credible metamorphic-testing pass-rate study for LLMs in ICL, but several metamorphic relations are over-strict and the 'bug' claims run ahead of the data.","tokens_in":11946,"tokens_out":3982,"would_cite":false,"duration_ms":40154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Software-testing-style metamorphic tests expose linguistic bugs in large language models that accuracy scores miss.","keywords":["in-context learning","metamorphic testing","LLM evaluation","adversarial perturbations","trustworthy AI","software testing","robustness","prompt engineering"],"falsifier":"Human annotators rate whether each perturbed item truly preserves the original meaning and sentiment on the paper's 50 sentiment items; if a large fraction is judged not meaning-preserving, or if a model can achieve near-100% pass rate on all nine relations while producing clearly wrong answers on the original inputs, then the pass-rate metric is measuring the relations themselves rather than trustworthiness.","tokens_in":10836,"feed_emoji":"🧪","tokens_out":6343,"duration_ms":57050,"temperature":0.7,"pith_summary":"The paper claims that the trustworthiness of in-context learning in large language models should be tested the way software is tested: by checking whether model outputs remain consistent under controlled input perturbations, not merely whether they match gold labels. It introduces MMT4NL, a metamorphic-testing framework that builds nine perturbation operators—synonym substitution, named-entity replacement, negation, vocabulary, fairness, robustness, temporal, semantic-role, and coreference—and measures a pass rate, the fraction of perturbed inputs for which the expected output relation holds. On 50 sentiment-analysis and 50 question-answering items, GPT-4o and Gemini-2.0-Flash achieve high accuracy on original inputs yet show substantially lower pass rates under many perturbations, such as Gemini-2.0-Flash's low temporal pass rate and negation pass rates as low as 60% in QA. The paper interprets these gaps as linguistic bugs in prompt interpretation that standard accuracy evaluation misses.","feed_headline":"Metamorphic tests reveal LLM bugs that accuracy misses","feed_subtitle":"GPT-4o and Gemini-2.0-Flash score high on benchmarks but flip answers under small linguistic changes.","key_machinery":"The central mechanism is the metamorphic relation, formally written as $MR(x, f(x), P(x), P(f(x))) \\Rightarrow g(P(x)) = P(f(x))$, where $P$ is the LLM under test, $f$ is an input transformation, and $g$ is the expected output transformation, usually the identity. It converts the test-oracle problem—no known correct output for a novel input—into a label-free consistency check between original and perturbed outputs. MMT4NL pairs this relation with nine concrete perturbation operators and a pass-rate metric $N_{\\text{pass}} / N_{\\text{test}}$, which lets any LLM API be tested without internal access or human annotation for each perturbed example.","core_discovery":"On the paper's own terms, the core discovery is that treating an LLM as a black-box program and applying metamorphic testing reveals systematic interpretation flaws that accuracy on original inputs conceals. The framework defines metamorphic relations as implications $P(x) = P(f(x))$ (or an analogous output relation) for meaning-preserving transformations $f$; when the equality fails, the model has failed a trustworthiness check. Across the two models and two tasks, the authors observe concrete failures: gender-modified sentences shift sentiment, synonym substitutions like 'complicated' to 'complex' flip negative sentiment to Neutral, a single transposed character in a medical term changes a QA answer, and coreference restructuring flips answers about census questions. Because accuracy remains high on the same items, the paper concludes that pass rate under metamorphic relations is a complementary, achievable trustworthiness metric for in-context learning.","pith_inferences":["The invariance standard is an editorial choice of the paper, not an external truth; a natural extension would be human validation of 'meaning preserved' for each perturbation to separate genuine semantic flips from acceptable re-readings.","The same metamorphic template could extend beyond classification to generation tasks by replacing equality with entailment or semantic-similarity output relations.","Pass-rate patterns across categories could be used as a diagnostic fingerprint—e.g., consistent Taxonomy failures with Fairness passes might point to embedding-space synonym drift rather than demographic bias.","Automating perturbation generation with controlled edit distance would scale the framework to large benchmarks and test whether pass rates degrade smoothly with perturbation severity."],"forward_implications":["Accuracy on original inputs should no longer be taken as sufficient evidence of ICL trustworthiness; pass rate under metamorphic relations should be reported alongside accuracy.","The framework supplies a reusable test template for any prompt construction: run the nine perturbation types, compute pass rates, and identify the specific linguistic capability that fails.","Few-shot prompting does not uniformly repair robustness: some categories, such as temporal consistency for Gemini-2.0-Flash and negation handling in QA for both models, remain weak or worsen.","Supplying context in question answering substantially raises both accuracy and pass rate, indicating that grounding can mitigate but not eliminate perturbation sensitivity."],"supporting_citations":[{"why":"Supplies the metamorphic testing method and the survey of metamorphic relations that the framework adapts to LLM evaluation.","marker":"[13]"},{"why":"Original work introducing metamorphic testing, the basis for the label-free oracle approach.","marker":"[3]"},{"why":"CheckList framework whose perturbation categories (taxonomy, NER, negation, robustness, etc.) the paper adapts into its nine test types.","marker":"[12]"},{"why":"PromptBench, the adversarial prompt-attack library that motivates combining linguistic interpretability with quantified attack success.","marker":"[16]"},{"why":"Survey of the test-oracle problem, the justification for using metamorphic relations rather than expected outputs.","marker":"[2]"},{"why":"Chat Sentiment dataset used for the sentiment analysis experiments.","marker":"[10]"},{"why":"StrategyQA dataset used for the question-answering experiments.","marker":"[14]"}],"fun_headline_variants":["Metamorphic testing finds what accuracy misses in LLMs","LLM trust tested via metamorphic relations, not just accuracy","Small language changes reveal LLM bugs that accuracy hides","Software-style metamorphic tests expose hidden LLM failures","Pass rates under metamorphic relations beat accuracy for LLM trust"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a trustworthy LLM must give invariant or predictably related outputs under meaning-preserving perturbations; if synonym and pronoun substitutions are not a valid standard of trustworthiness, then low pass rates do not prove the model is buggy.","fun_headline_variants_meta":{"raw":{"variants":["Metamorphic testing finds what accuracy misses in LLMs","LLM trust tested via metamorphic relations, not just accuracy","Small language changes reveal LLM bugs that accuracy hides","Software-style metamorphic tests expose hidden LLM failures","Pass rates under metamorphic relations beat accuracy for LLM trust"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001354,"raw_usage":{"total_tokens":5476,"prompt_tokens":905,"completion_tokens":4571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":4490}},"tokens_in":521,"tokens_out":4571,"duration_ms":31730,"temperature":1.0,"reasoning_tokens":4490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:07:52.126259+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Human annotators rate whether each perturbed item truly preserves the original meaning and sentiment on the paper's 50 sentiment items; if a large fraction is judged not meaning-preserving, or if a model can achieve near-100% pass rate on all nine relations while producing clearly wrong answers on the original inputs, then the pass-rate metric is measuring the relations themselves rather than trustworthiness.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original work introducing metamorphic testing, the basis for the label-free oracle approach."},{"cited_title":"In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"CheckList framework whose perturbation categories (taxonomy, NER, negation, robustness, etc.) the paper adapts into its nine test types."},{"cited_title":"Kaggle (2023), available athttps://www","cited_arxiv_id":null,"evidence_quote":"Chat Sentiment dataset used for the sentiment analysis experiments."},{"cited_title":"Hugging Face Datasets (2023), available athttps:// huggingface.co/datasets/voidful/StrategyQA","cited_arxiv_id":null,"evidence_quote":"StrategyQA dataset used for the question-answering experiments."}],"review_version":1}