{"id":"94093caf-3e12-48d8-ab5f-e24e1e8bdc4b","arxiv_id":"2412.15557","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MORTAR uses dialogue-level perturbations and metamorphic relations to reveal context-dependent bugs in multi-turn LLM dialogue systems without an LLM judge.","lead":"This paper introduces MORTAR, a testing method for multi-turn chatbot conversations that generates modified dialogue histories and checks whether the chatbot's answers stay consistent. In tests on six small open-source LLMs, it reported over 150% more bugs per test case than a single-turn comparison method, though no code or statistical replication details are provided.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EC, the automated equivalent-context check, is the load-bearing component: the paper's own 'How?' example shows inequivalent contexts mislabelled as equivalent, the error direction that inflates the dominant MR1/MR3 detections, and no ground-truth accuracy for EC is provided.","rationale":"The paper makes a real contribution: it formalises multi-turn metamorphic relations, proposes five dialogue-level perturbations, and provides an automated MR-matching mechanism. I credit the framework and the ablation study as genuine evidence of design value. However, the empirical headline rests on counting MR violations, and that count is only meaningful if the equivalent-context check is reliable. The authors themselves flag the weak direction in Section III-E and provide a concrete failure in Section V-D2; the reported precision estimate is based on a small sample and does not isolate EC accuracy. My stress test is therefore not an objection to the framework's novelty, but to the evidential status of the 150% and 153% effectiveness claims, which are point estimates without significance tests, error bars, or released code and data. If the proposed manual EC validation changes the counts materially, the paper's central claim would need to be weakened to a framework proposal with preliminary evidence; if the counts are stable, the claim stands. This is exactly the CONDITIONAL posture the reader adopted, so my read does not change the verdict.","tokens_in":25422,"tokens_out":6391,"duration_ms":61539,"concrete_test":"Take a stratified random sample of 300 (perturbed context, target question) pairs from MORTAR's 2015 generated test cases (60 per perturbation), and have two annotators independently decide, using the original CoQA story and dialogue, whether the perturbed context is informationally equivalent to the original context for that target question. Restrict the MR1 and MR3 violation counts to pairs where both annotators classify EC=True, recompute NBugs, RETC and BPTC for each of the six SUTs, and compare the adjusted BPTC values to METAL's reported BPTC values. If the adjusted MORTAR advantage falls below ~150% or becomes non-significant under a paired bootstrap over dialogues, the central effectiveness claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-E defines EC and admits the assumption 'might exist limitations for MR1 and MR3, as some inequivalent contexts will be mislabelled as equivalent'; Section V-D2 then documents a concrete instance: 'How?' is asked after 'Did the movie break any records?' in the perturbed test case, while the original context was 'When did Kyle die?', and EC fails to flag the changed referent. This is exactly the error direction that matters. MR1 and MR3 together account for about 99.5% of MORTAR's reported bug detections (Table VIII means: MR1 18,326; MR3 4,451; MR2 8; MR4 108), and both require trusted EC=True classifications. Any systematic tendency to label inequivalent contexts as equivalent therefore inflates NBugs, RETC, and BPTC directly. The reported PPD of 70.5% comes from only 100 manually labelled detections, with no confidence interval and no EC-specific accuracy measurement against human equivalence judgements; the one traceable false positive goes in the inflation-biased direction. Thus the headline claim that MORTAR finds 'over 150% more bugs per test case' than METAL is not yet supported until the dominant MR1/MR3 detections are re-derived under a validated EC.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MORTAR, a metamorphic testing approach for multi-turn question-answer dialogue systems based on LLMs. It formalizes a multi-turn metamorphic testing framework, defines five dialogue-level perturbations (DRS, DRR, DRD, DSR, DSD), four metamorphic relations (MR1-MR4), and an automated equivalent-context check that combines semantic, ontology-based, and dataset-specific checks. Using CoQA as the test seed source, the authors compare MORTAR against the single-turn metamorphic testing baseline METAL on six open-source dialogue systems, reporting that MORTAR detects over 150% more bugs per test case and achieves higher precision, diversity, and uniqueness of bugs. An ablation study attributes contributions to individual perturbations and MRs.","tokens_in":25758,"tokens_out":3255,"duration_ms":30579,"significance":"If the equivalent-context check is reliable, MORTAR is a meaningful contribution: it is a fully automated, judge-free multi-turn metamorphic testing method with a formalized framework, novel dialogue-level perturbations, and automated MR matching. The paper ships a substantial empirical study with six dialogue systems and an ablation analysis, which strengthens the practical relevance of the proposal. The main significance, however, is conditional on the validity of EC, because MR1 and MR3, which together account for about 99.5% of reported detections (Table VIII), both rely on trusting EC labels of context equivalence. The authors' own discussion in Section V-D2 documents a concrete failure of EC in the direction that inflates MR1/MR3 detections, and no independent accuracy evaluation of EC is provided. Thus the empirical claims are plausible but not yet established to the standard required for the headline effectiveness conclusions.","major_comments":[{"comment":"The automated equivalent-context check is the load-bearing component of MORTAR, yet its accuracy is never measured against ground truth. Section III-E explicitly states that 'some inequivalent contexts will be mislabelled as equivalent' for MR1 and MR3, and Section V-D2 gives a concrete instance: in a perturbed test case, 'How?' is asked after 'Did the movie break any records?' while in the original test case 'How?' followed 'When did Kyle die?', and EC fails to flag the changed referent. This is exactly the error direction that inflates detections: MR1 and MR3 together account for 22,777 of the 22,893 mean detections in Table VIII (about 99.5%), and both require trusting EC=True classifications. The reported PPD of 70.5% is based on only 100 manually labelled detections and does not measure EC-specific precision or recall. The authors should validate EC against human equivalence judgments on a sample of perturbed contexts, report precision/recall for the EC=True and EC=False directions separately, and re-derive NBugs, RETC, BPTC, and PPD after filtering or correcting known EC failures.","section":"Section III-E and V-D2"},{"comment":"The comparison between MORTAR and METAL is asymmetric with respect to unanswerable questions. Section IV-C states that MORTAR excludes unanswerable questions from MR violation detection, while Section IV-D states that METAL treats all originally unanswerable questions as answerable with the expectation 'Unknown'. This differential handling directly affects the counts that drive the headline comparison (NBugs, Rate+, BPTC). The authors should report the number of unanswerable questions in the test set, and either re-run the analysis with consistent treatment or provide a sensitivity analysis showing that the 150% effectiveness advantage is robust to this asymmetry.","section":"Section IV-C and IV-D"},{"comment":"The effectiveness comparison is based on non-overlapping test seed sets: METAL uses all 500 test seeds while MORTAR uses only 403 successfully processed seeds (Table III). Section V-D1 reports that the failed extraction dialogues overlap 92.8% with effective original test cases of DS6, the best-performing system, which creates a systematic selection bias in favor of MORTAR. In addition, no statistical significance tests or confidence intervals are reported for any of the effectiveness metrics, despite the abstract claiming results are 'significantly better'. The authors should compare both methods on the common 403-seed subset, report the results for the excluded seeds separately, and provide significance tests or confidence intervals for the key metrics (NBugs, BPTC, Rate+, PPD).","section":"Section V-A and V-D1"}],"minor_comments":[{"comment":"In Table IV, the DS1 row for METAL shows '01.023' for CV, which appears to be a typo for '1.023'; please verify and correct.","section":"Table IV"},{"comment":"The prompt description says the dialogue systems are required to answer 'Unknow' if they do not know the answer; this should be 'Unknown'.","section":"Section IV-E"},{"comment":"The heading 'Dataset-specified check' is fine, but the phrase 'an additional semantic-based check will be added' is slightly confusing because this check is described as separate from the earlier semantic-based check; please clarify how it differs.","section":"Section III-E1c"},{"comment":"The description of manual labelling says cases with a score below six are classified as true positive detections, but the treatment of exactly six is unspecified; please clarify the threshold rule.","section":"Section V-B2"},{"comment":"In Table VI, the MR3 row shows the same value (6,194) for all five perturbations, but MR3 is a group-level relation; the per-perturbation attribution is not obvious. Please explain how the counting is performed for MR3 and MR4.","section":"Table VI"},{"comment":"Equation (27) defines EC on (Q^r_{i-1}, q^r_i), while Equation (30) uses EC(Q_{j-1}, q_j) = True; the two-argument notation for the original context should be defined explicitly for consistency.","section":"Section III-D and III-E"},{"comment":"The sentence 'This contradicts the trend of increased reasoning capability as expectations' is grammatically unclear; please revise to 'as expected'.","section":"Section V-D3"}],"recommendation":"major_revision","confidential_remarks":"The paper does not mention an artifact or replication package. Given that the core claims depend on an automated tool (EC) and a pipeline of LLM-based information extraction, making code, prompts, and processed data available would substantially help reviewers and future users verify the results. I would encourage the editor to ask for this as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: MORTAR is worth engaging with. The multi-turn metamorphic framework — round-level perturbations and four MRs with an automated equivalence check — is a genuine step past single-turn METAL-style testing. For that reason I'd send it out. But the headline claim, 'over 150% more bugs per test case,' is not yet supported.\n\nWhat's new: dialogue-level operations (shuffle, reduction, duplication, and their combinations) are a real move beyond word/character perturbations. The formalization of MR1–MR4 and the idea of using an automated context-equivalence check to match MRs dynamically is clever and avoids LLM judges. The ablation study is a plus.\n\nWhere it's soft: the whole edifice leans on EC, the equivalent-context check. The paper admits (Section III-E) that some inequivalent contexts will be mislabelled as equivalent, and Section V-D2 gives a concrete case ('How?' after 'Did the movie break any records?' vs. original 'When did Kyle die?') where EC fails in exactly the direction that inflates MR1/MR3 detections. Those two relations account for ~99.5% of reported bugs. Without a ground-truth accuracy measurement of EC against human equivalence judgments, the bug counts are essentially unvalidated. Also, only 403 of 500 dialogues were processed, and the failures overlap 92.8% with the effective DS6 test cases, so the comparison is biased against the baseline. No significance tests or error bars anywhere, and PPD of 70.5% comes from 100 manually labelled samples. That's a point estimate with no interval.\n\nNone of these are fatal to the framework. They are fixable: validate EC on a labelled sample, re-run the numbers only for cases where EC is trustworthy, report uncertainty, and ship code/data. But as written, the evidence doesn't support the strength of the claims.\n\nWho this is for: people working on QA metamorphic testing for LLM systems will find the formalization and perturbations useful, even if they ignore the empirical section. I'd bring it to a reading group, but with a critical eye on the EC evaluation.\n\nRecommendation: serious referee, major revision. I'd accept the paper if the re-analysis under a validated EC still shows a meaningful advantage over METAL; if the EC-dependent numbers collapse, the contribution shrinks to a framework paper, which is still publishable but with much weaker claims.","headline":"MORTAR's multi-turn MR framework is a real contribution, but the EC check that powers it is unvalidated and the empirical claims need stronger support.","tokens_in":26257,"tokens_out":2627,"would_cite":false,"duration_ms":22757,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MORTAR tests multi-turn LLM dialogues by perturbing context, finding over 150% more bugs per test case than single-turn metamorphic testing.","keywords":["metamorphic testing","multi-turn dialogue","LLM-based dialogue systems","test oracle problem","dialogue-level perturbation","equivalent context check","automated testing","bug detection"],"falsifier":"Manually annotate context equivalence on a sample of perturbed dialogues (e.g., 200 target questions where the target is a short anaphoric question like 'How?'), then compare EC's decisions to human labels. If EC labels a large share of inequivalent contexts as equivalent (say, above 25–30%), then MR1-violation detections on those cases are false positives and the claimed bug counts and precision would drop accordingly.","tokens_in":25262,"feed_emoji":"🧪","tokens_out":5349,"duration_ms":42811,"temperature":0.7,"pith_summary":"MORTAR is a metamorphic testing method that targets LLM-based dialogue systems in multi-turn conversations, where most real-world use happens but testing tools are scarce. It generates follow-up dialogues by perturbing the conversational context—shuffling, deleting, or duplicating rounds—and decides automatically whether each perturbed context is informationally equivalent to the original. On that decision it selects one of four metamorphic relations that specify what a correct dialogue system should output. In experiments on six open-source dialogue systems, MORTAR reports over 150% more bugs per test case than the single-turn metamorphic baseline METAL, with higher precision and a larger share of unique bugs. The paper's contribution is a formal framework that turns multi-turn dialogue testing into an automated, judge-free process.","feed_headline":"Multi-turn context tests find 2.5x more LLM dialogue bugs","feed_subtitle":"Shuffling, deleting, and duplicating dialogue rounds exposes bugs without needing an LLM judge.","key_machinery":"The equivalent-context check (EC), which decides whether a perturbed dialogue context supplies the same information as the original context for a given target question. EC combines three automated checks—semantic-based anaphora resolution, ontology-based knowledge-graph comparison, and a dataset-specific check—and its True/False output routes every question to the appropriate metamorphic relation (MR1/MR3 when equivalent, MR2/MR4 when not). The automated MR matching made possible by EC is what allows MORTAR to be fully automated and judge-free.","core_discovery":"The central claim is that the test-oracle problem in multi-turn dialogue testing can be mitigated without human labels or LLM judges by exploiting the context-dependence of dialogue. MORTAR formalises a multi-turn dialogue as a sequence of question-answer pairs, introduces five dialogue-level perturbations (round shuffle, round reduction, round duplication, and two combined variants) that alter the context of target questions, and defines four metamorphic relations: MR1 expects semantically similar answers when the perturbed context is equivalent to the original, MR2 expects different answers when the context loses critical information, MR3 requires consistency among all context-preserving versions, and MR4 requires divergence between context-preserving and context-altering versions. The equivalence decision is automated by an equivalent-context check built from semantic anaphora resolution, an ontology/graph-based information extraction pipeline, and a dataset-specific check. On six open-source LLM-based dialogue systems using CoQA as test seeds, MORTAR reports 36,908 positive detections versus 14,665 for METAL, a BPTC of 11.36 versus 4.49, a manual-check precision of 70.5% versus 45.5%, and 35.6% unique bugs versus 25.9%.","pith_inferences":["Improving EC accuracy with a stronger context-equivalence model would likely raise MORTAR's effectiveness further, since the paper's own error analysis shows EC's main weakness is mislabelling inequivalent contexts as equivalent.","The MR template could be lifted to other multi-turn properties, such as safety or factuality: context-preserving perturbations should not change a system's refusal, and context-altering perturbations should not cause it to keep asserting previously grounded facts.","The ontology-based check's reliance on LLM-based information extraction makes MORTAR's seed yield (403/500 dialogues) and precision sensitive to the IE pipeline; domain-specific dialogues with complex entities would require adapted extraction prompts.","The L3-bug category—failures found in dialogues that passed reference-based testing—is a reusable metric for any metamorphic testing of conversational systems and could become a standard reporting statistic."],"forward_implications":["Existing single-turn QA datasets can be reused as multi-turn test seeds, amplifying bug discovery without new annotated oracles.","Adding new dialogue-level perturbations only requires an EC that can judge their effect; the four MRs and MR-matching mechanism remain intact.","Bugs that reference-based testing misses (L2 and L3 bugs) become accessible from the same test seeds, so resource-constrained testing covers more failure modes.","Because no LLM judge is involved, the reported bugs rest on formal metamorphic relations rather than on the opinions of another model.","The method's smaller performance drop across six systems (51.8 vs 57.7 percentage points in positive rate) indicates more consistent effectiveness when dialogue systems vary in quality."],"supporting_citations":[{"why":"The single-turn metamorphic testing framework METAL, which serves as the baseline that MORTAR claims to outperform by over 150% in bugs per test case.","marker":"[16]"},{"why":"The CoQA multi-turn reading-comprehension dataset used as the test seeds for all experiments.","marker":"[18]"},{"why":"The original metamorphic testing proposal whose oracle-free principle MORTAR extends to multi-turn dialogue.","marker":"[13]"},{"why":"The extract-define-canonicalize framework that inspires MORTAR's LLM-based knowledge-graph construction for the ontology-based check.","marker":"[41]"},{"why":"The GraphRAG approach that inspires the LLM pipeline for extracting dialogue entities and relations.","marker":"[40]"},{"why":"Evidence cited for why LLM judges are biased, motivating MORTAR's judge-free design.","marker":"[12]"},{"why":"A survey of metamorphic testing that situates MORTAR within established MT practice and supports the claim that MT mitigates the oracle problem.","marker":"[30]"}],"fun_headline_variants":["Multi-turn metamorphic tests find 2.5x more dialogue bugs","Automated multi-turn testing exposes 2.5x more dialogue bugs","Multi-turn context changes reveal 2.5x more LLM dialogue bugs","No LLM judge needed: multi-turn tests find 2.5x bugs","Shuffling dialogue turns uncovers 2.5x more bugs in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated equivalent-context check correctly decides whether a perturbed context preserves the information needed to answer the target question; if it mislabels inequivalent contexts as equivalent, many reported bugs are false positives (as the paper's 'How?' example shows).","fun_headline_variants_meta":{"raw":{"variants":["Multi-turn metamorphic tests find 2.5x more dialogue bugs","Automated multi-turn testing exposes 2.5x more dialogue bugs","Multi-turn context changes reveal 2.5x more LLM dialogue bugs","No LLM judge needed: multi-turn tests find 2.5x bugs","Shuffling dialogue turns uncovers 2.5x more bugs in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2707,"prompt_tokens":1061,"completion_tokens":1646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":1544}},"tokens_in":677,"tokens_out":1646,"duration_ms":10242,"temperature":1.0,"reasoning_tokens":1544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:19:19.864873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate context equivalence on a sample of perturbed dialogues (e.g., 200 target questions where the target is a short anaphoric question like 'How?'), then compare EC's decisions to human labels. If EC labels a large share of inequivalent contexts as equivalent (say, above 25–30%), then MR1-violation detections on those cases are false positives and the claimed bug counts and precision would drop accordingly.","supporting_citations":[{"cited_title":"Metal: Metamorphic testing framework for analyzing large-language model qualities,","cited_arxiv_id":null,"evidence_quote":"The single-turn metamorphic testing framework METAL, which serves as the baseline that MORTAR claims to outperform by over 150% in bugs per test case."},{"cited_title":"Coqa: A conversational question answering challenge,","cited_arxiv_id":null,"evidence_quote":"The CoQA multi-turn reading-comprehension dataset used as the test seeds for all experiments."},{"cited_title":"Appli- cation of metamorphic testing in numerical analysis,","cited_arxiv_id":null,"evidence_quote":"The original metamorphic testing proposal whose oracle-free principle MORTAR extends to multi-turn dialogue."}],"review_version":1}