Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems

T0 review · 3 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MORTAR tests multi-turn LLM dialogues by perturbing context, finding over 150% more bugs per test case than single-turn metamorphic testing.

desk verdict MORTAR's multi-turn MR framework is a real contribution, but the EC check that powers it is unvalidated and the empirical claims need stronger support. read the letter →

arxiv 2412.15557 v4 pith:AQFIRS6X submitted 2024-12-20 cs.SE cs.CL

Aaron Guoxiang Guo , Aldeida Aleti , Neelofar Neelofar , Chakkrit Tantithamthavorn , Yuanyuan Qi , Tsong Yueh Chen This is my paper · ORCID
classification cs.SEcs.CL
keywords metamorphictestingmulti-turndialogueLLM-basedsystemstestoracleproblemdialogue-levelperturbationequivalentcontextcheckautomatedbugdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MORTAR is a metamorphic testing method that targets LLM-based dialogue systems in multi-turn conversations, where most real-world use happens but testing tools are scarce. It generates follow-up dialogues by perturbing the conversational context—shuffling, deleting, or duplicating rounds—and decides automatically whether each perturbed context is informationally equivalent to the original. On that decision it selects one of four metamorphic relations that specify what a correct dialogue system should output. In experiments on six open-source dialogue systems, MORTAR reports over 150% more bugs per test case than the single-turn metamorphic baseline METAL, with higher precision and a larger share of unique bugs. The paper's contribution is a formal framework that turns multi-turn dialogue testing into an automated, judge-free process.

What carries the argument

The equivalent-context check (EC), which decides whether a perturbed dialogue context supplies the same information as the original context for a given target question. EC combines three automated checks—semantic-based anaphora resolution, ontology-based knowledge-graph comparison, and a dataset-specific check—and its True/False output routes every question to the appropriate metamorphic relation (MR1/MR3 when equivalent, MR2/MR4 when not). The automated MR matching made possible by EC is what allows MORTAR to be fully automated and judge-free.

What would settle it

Manually annotate context equivalence on a sample of perturbed dialogues (e.g., 200 target questions where the target is a short anaphoric question like 'How?'), then compare EC's decisions to human labels. If EC labels a large share of inequivalent contexts as equivalent (say, above 25–30%), then MR1-violation detections on those cases are false positives and the claimed bug counts and precision would drop accordingly.

Watch

Extended reading notes

Core claim

The central claim is that the test-oracle problem in multi-turn dialogue testing can be mitigated without human labels or LLM judges by exploiting the context-dependence of dialogue. MORTAR formalises a multi-turn dialogue as a sequence of question-answer pairs, introduces five dialogue-level perturbations (round shuffle, round reduction, round duplication, and two combined variants) that alter the context of target questions, and defines four metamorphic relations: MR1 expects semantically similar answers when the perturbed context is equivalent to the original, MR2 expects different answers when the context loses critical information, MR3 requires consistency among all context-preserving versions, and MR4 requires divergence between context-preserving and context-altering versions. The equivalence decision is automated by an equivalent-context check built from semantic anaphora resolution, an ontology/graph-based information extraction pipeline, and a dataset-specific check. On six open-source LLM-based dialogue systems using CoQA as test seeds, MORTAR reports 36,908 positive detections versus 14,665 for METAL, a BPTC of 11.36 versus 4.49, a manual-check precision of 70.5% versus 45.5%, and 35.6% unique bugs versus 25.9%.

Load-bearing premise

The load-bearing premise is that the automated equivalent-context check correctly decides whether a perturbed context preserves the information needed to answer the target question; if it mislabels inequivalent contexts as equivalent, many reported bugs are false positives (as the paper's 'How?' example shows).

Editorial extensions

If this is right

  • Existing single-turn QA datasets can be reused as multi-turn test seeds, amplifying bug discovery without new annotated oracles.
  • Adding new dialogue-level perturbations only requires an EC that can judge their effect; the four MRs and MR-matching mechanism remain intact.
  • Bugs that reference-based testing misses (L2 and L3 bugs) become accessible from the same test seeds, so resource-constrained testing covers more failure modes.
  • Because no LLM judge is involved, the reported bugs rest on formal metamorphic relations rather than on the opinions of another model.
  • The method's smaller performance drop across six systems (51.8 vs 57.7 percentage points in positive rate) indicates more consistent effectiveness when dialogue systems vary in quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Improving EC accuracy with a stronger context-equivalence model would likely raise MORTAR's effectiveness further, since the paper's own error analysis shows EC's main weakness is mislabelling inequivalent contexts as equivalent.
  • The MR template could be lifted to other multi-turn properties, such as safety or factuality: context-preserving perturbations should not change a system's refusal, and context-altering perturbations should not cause it to keep asserting previously grounded facts.
  • The ontology-based check's reliance on LLM-based information extraction makes MORTAR's seed yield (403/500 dialogues) and precision sensitive to the IE pipeline; domain-specific dialogues with complex entities would require adapted extraction prompts.
  • The L3-bug category—failures found in dialogues that passed reference-based testing—is a reusable metric for any metamorphic testing of conversational systems and could become a standard reporting statistic.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes MORTAR, a metamorphic testing approach for multi-turn question-answer dialogue systems based on LLMs. It formalizes a multi-turn metamorphic testing framework, defines five dialogue-level perturbations (DRS, DRR, DRD, DSR, DSD), four metamorphic relations (MR1-MR4), and an automated equivalent-context check that combines semantic, ontology-based, and dataset-specific checks. Using CoQA as the test seed source, the authors compare MORTAR against the single-turn metamorphic testing baseline METAL on six open-source dialogue systems, reporting that MORTAR detects over 150% more bugs per test case and achieves higher precision, diversity, and uniqueness of bugs. An ablation study attributes contributions to individual perturbations and MRs.

Significance. If the equivalent-context check is reliable, MORTAR is a meaningful contribution: it is a fully automated, judge-free multi-turn metamorphic testing method with a formalized framework, novel dialogue-level perturbations, and automated MR matching. The paper ships a substantial empirical study with six dialogue systems and an ablation analysis, which strengthens the practical relevance of the proposal. The main significance, however, is conditional on the validity of EC, because MR1 and MR3, which together account for about 99.5% of reported detections (Table VIII), both rely on trusting EC labels of context equivalence. The authors' own discussion in Section V-D2 documents a concrete failure of EC in the direction that inflates MR1/MR3 detections, and no independent accuracy evaluation of EC is provided. Thus the empirical claims are plausible but not yet established to the standard required for the headline effectiveness conclusions.

major comments (3)
  1. [Section III-E and V-D2] The automated equivalent-context check is the load-bearing component of MORTAR, yet its accuracy is never measured against ground truth. Section III-E explicitly states that 'some inequivalent contexts will be mislabelled as equivalent' for MR1 and MR3, and Section V-D2 gives a concrete instance: in a perturbed test case, 'How?' is asked after 'Did the movie break any records?' while in the original test case 'How?' followed 'When did Kyle die?', and EC fails to flag the changed referent. This is exactly the error direction that inflates detections: MR1 and MR3 together account for 22,777 of the 22,893 mean detections in Table VIII (about 99.5%), and both require trusting EC=True classifications. The reported PPD of 70.5% is based on only 100 manually labelled detections and does not measure EC-specific precision or recall. The authors should validate EC against human equivalence judgments on a sample of perturbed contexts, report precision/recall for the EC=True and EC=False directions separately, and re-derive NBugs, RETC, BPTC, and PPD after filtering or correcting known EC failures.
  2. [Section IV-C and IV-D] The comparison between MORTAR and METAL is asymmetric with respect to unanswerable questions. Section IV-C states that MORTAR excludes unanswerable questions from MR violation detection, while Section IV-D states that METAL treats all originally unanswerable questions as answerable with the expectation 'Unknown'. This differential handling directly affects the counts that drive the headline comparison (NBugs, Rate+, BPTC). The authors should report the number of unanswerable questions in the test set, and either re-run the analysis with consistent treatment or provide a sensitivity analysis showing that the 150% effectiveness advantage is robust to this asymmetry.
  3. [Section V-A and V-D1] The effectiveness comparison is based on non-overlapping test seed sets: METAL uses all 500 test seeds while MORTAR uses only 403 successfully processed seeds (Table III). Section V-D1 reports that the failed extraction dialogues overlap 92.8% with effective original test cases of DS6, the best-performing system, which creates a systematic selection bias in favor of MORTAR. In addition, no statistical significance tests or confidence intervals are reported for any of the effectiveness metrics, despite the abstract claiming results are 'significantly better'. The authors should compare both methods on the common 403-seed subset, report the results for the excluded seeds separately, and provide significance tests or confidence intervals for the key metrics (NBugs, BPTC, Rate+, PPD).
minor comments (7)
  1. [Table IV] In Table IV, the DS1 row for METAL shows '01.023' for CV, which appears to be a typo for '1.023'; please verify and correct.
  2. [Section IV-E] The prompt description says the dialogue systems are required to answer 'Unknow' if they do not know the answer; this should be 'Unknown'.
  3. [Section III-E1c] The heading 'Dataset-specified check' is fine, but the phrase 'an additional semantic-based check will be added' is slightly confusing because this check is described as separate from the earlier semantic-based check; please clarify how it differs.
  4. [Section V-B2] The description of manual labelling says cases with a score below six are classified as true positive detections, but the treatment of exactly six is unspecified; please clarify the threshold rule.
  5. [Table VI] In Table VI, the MR3 row shows the same value (6,194) for all five perturbations, but MR3 is a group-level relation; the per-perturbation attribution is not obvious. Please explain how the counting is performed for MR3 and MR4.
  6. [Section III-D and III-E] Equation (27) defines EC on (Q^r_{i-1}, q^r_i), while Equation (30) uses EC(Q_{j-1}, q_j) = True; the two-argument notation for the original context should be defined explicitly for consistency.
  7. [Section V-D3] The sentence 'This contradicts the trend of increased reasoning capability as expectations' is grammatically unclear; please revise to 'as expected'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MORTAR's bug counts are empirical and externally checked; EC oracle-leakage is a validity threat, not a definitional reduction.

full rationale

MORTAR's central effectiveness claims (NBugs, RETC, BPTC) are empirical counts obtained by running six dialogue systems on perturbed CoQA dialogues and comparing with the METAL baseline; no fitted constant is later renamed as a prediction. The four metamorphic relations are stated assumptions (Eqs. 30-39) linking perturbed contexts to expected answer similarity, and the equivalent-context check is an automated heuristic rather than a parameter fitted to the reported bug counts. The paper explicitly concedes EC limitations in Section III-E and gives a concrete misclassification example in Section V-D2, and the manual PPD assessment in Section V-B2 provides an external, human-labelled check on whether reported detections are true positives. The use of original answers in the ontology-based check is a possible oracle-leakage and validity threat, but it does not make the bug counts equivalent by construction: a bug is counted only when the dialogue system's actual output violates the MR, so the results remain empirically grounded. Self-citations, such as Chen et al.'s original metamorphic testing work [13] and Aleti [2], serve as background references and are not load-bearing uniqueness arguments. No step in the paper reduces to its own input by definition.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the assumptions that answers in CoQA are unique and depend only on sufficient context, that the equivalent-context check can be automated with the three heuristics, and that semantic similarity above 0.6 with all-MiniLM-L6-v2 captures answer equivalence. The original seed answers are used inside the context check, which injects oracle information before MR1 expects the output to match that oracle. No new physical entities are introduced; the L1/L2/L3 bug taxonomy and MRs are methodological constructs, not entities.

free parameters (4)
  • semantic similarity threshold epsilon = 0.6
    Hand-set threshold equal to the METAL baseline (Section IV-H). It defines every MR violation, so the number and precision of bugs directly depend on it.
  • reduction ratio for DRR and DSR = 30%
    Randomly chosen proportion of rounds removed (Section IV-H); governs how much context is altered.
  • duplication ratio for DRD and DSD = 20%
    Randomly chosen proportion of rounds duplicated (Section IV-H); governs test case length and context redundancy.
  • manual precision score threshold = score below 6 is a true positive
    Human raters score dialogue system outputs from 0 to 10; outputs scoring below 6 are classified as true positive detections in the precision evaluation (Section V-D2). This cut point is arbitrary.
assumptions (6)
  • domain assumption Each multi-turn QA question has a unique correct answer given sufficient context
    Used to define the ideal dialogue system and the equivalence relation in Eqs. (7)-(18); this excludes open-ended or opinion questions.
  • domain assumption The answer a_i is implied by the question sequence Q_i alone (Eq. 11)
    Used to rewrite IDS contexts without the CoQA story or document; the dataset-specified check later adds external story material, so the formal model omits a primary input of the experiments.
  • ad hoc to paper Equivalent context can be decided by anaphora resolution, knowledge-graph coverage, and a dataset-specific story check
    Load-bearing for MR matching; Section III-E admits the assumption may mislabel inequivalent contexts as equivalent, and Section V-D2 gives a concrete false-positive example.
  • ad hoc to paper Original answers from the seed dataset may be used inside the context check to clarify what information questions imply
    Section III-E states answers from original test cases are used to assist the analysis. This injects oracle information into the equivalence decision before MR1 expects the output to match that oracle.
  • domain assumption Semantic similarity above 0.6 measured by all-MiniLM-L6-v2 means equivalent answers
    All MR violations and RBT comparisons use this threshold; Section V-D2 notes that overlong but correct answers are flagged as bugs.
  • domain assumption Randomly chosen deletion, duplication, and shuffle operations produce representative test cases
    No random seeds or repeated runs are reported, and Section VI-A admits the perturbations lack fine-grained control.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems." pith.science (2026). https://pith.science/paper/AQFIRS6X

@misc{pith2026241215557,
  author       = {Pith},
  title        = {Pith review of: MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQFIRS6X}},
  note         = {Machine review of arXiv:2412.15557}
}
read the original abstract

With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in single-turn testing scenarios. However, multi-turn interaction is the common real-world usage of dialogue systems, yet testing methods for such interactions remain underexplored. This is largely due to the oracle problem in multi-turn testing, which continues to pose a significant challenge for dialogue system developers and researchers. In this paper, we propose MORTAR, a metamorphic multi-turn dialogue testing approach, which mitigates the test oracle problem in testing LLM-based dialogue systems. MORTAR formalises the multi-turn testing for dialogue systems, and automates the generation of question-answer dialogue test cases with multiple dialogue-level perturbations and metamorphic relations (MRs). The automated MR matching mechanism allows MORTAR more flexibility and efficiency in metamorphic testing. The proposed approach is fully automated without reliance on LLM judges. In testing six popular LLM-based dialogue systems, MORTAR reaches significantly better effectiveness with over 150\% more bugs revealed per test case when compared to the single-turn metamorphic testing baseline. Regarding the quality of bugs, MORTAR reveals higher-quality bugs in terms of diversity, precision and uniqueness. MORTAR is expected to inspire more multi-turn testing approaches, and assist developers in evaluating the dialogue system performance more comprehensively with constrained test resources and budget.

Figures

Figures reproduced from arXiv: 2412.15557 by the authors.

Figure 1
Figure 1. Testing with different test cases. The test seed is the original test case [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview workflow of MORTAR. processing to judge the actual effect on the test seeds. It is necessary yet difficult to tell whether a target question in the perturbed test case is given sufficient information from context to be answerable. This brings an additional challenge to multi￾turn MT. Dialogue-level perturbation has not been proposed for its intrinsic difficulty in modelling and implementation. In summary… view at source ↗
Figure 3
Figure 3. Perturbation and MR violation detection. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: In ontology-based check, as the dialogue proceeds, information in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Information Extraction Procedure with LLM pipelines [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol

    cs.SE 2025-08 conditional novelty 5.0 of 10

    This position paper classifies testing methods for LLM applications into three layers and proposes AICL, a structured protocol for testable agent communication; neither the framework nor the protocol is empirically validated.

Reference graph

Works this paper leans on

56 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    A review of dialogue systems: current trends and future directions,

    A. Algherairy and M. Ahmed, “A review of dialogue systems: current trends and future directions,”Neural Computing and Applications, vol. 36, no. 12, pp. 6325–6351, 2024

    work page 2024

  2. [2]

    Software testing of generative ai systems: Challenges and opportunities,

    A. Aleti, “Software testing of generative ai systems: Challenges and opportunities,” in2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, pp. 4–14

    work page 2023

  3. [3]

    Hotpotqa: A dataset for diverse, explainable multi- hop question answering,

    Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi- hop question answering,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 2369– 2380

    work page 2018

  4. [4]

    Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

    B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaefferet al., “Decodingtrust: A comprehensive assessment of trustworthiness in gpt models.” inNeurIPS, 2023

    work page 2023

  5. [5]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

  6. [6]

    Gpqa: A graduate-level google-proof q&a benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level google-proof q&a benchmark,”arXiv preprint arXiv:2311.12022, 2023

    arXiv 2023

  7. [7]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wanget al., “A survey on evaluation of large language models,”ACM Transactions on Intelligent Systems and Technology, vol. 15, no. 3, pp. 1–45, 2024

    2024

  8. [8]

    ShareGPT Data,

    ShareGPT, “ShareGPT Data,” 2023, accessed: 2024-10-29. [Online]. Available: https://huggingface.co/datasets/anon8231489123/ShareGPT Vicuna unfiltered

    work page 2023

Show all 56 references
  1. [9]

    Llm defenses are not robust to multi-turn human jailbreaks yet,

    N. Li, Z. Han, I. Steneker, W. Primack, R. Goodside, H. Zhang, Z. Wang, C. Menghini, and S. Yue, “Llm defenses are not robust to multi-turn human jailbreaks yet,”arXiv preprint arXiv:2408.15221, 2024

    2024 arXiv

  2. [10]

    MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues,

    G. Bai, J. Liu, X. Bu, Y . He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, and W. Ouyang, “MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues,” in Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024

  3. [11]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”Advances in Neural Information Processing Systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  4. [13]

    Appli- cation of metamorphic testing in numerical analysis,

    F. Chan, T. Y . Chen, S. C. Cheung, M. F. Lau, and S.-M. Yiu, “Appli- cation of metamorphic testing in numerical analysis,” inProceedings of the IASTED International Conference on Software Engineering (SE’98), 1998

    1998

  5. [14]

    Testing your question answering software via asking recursively,

    S. Chen, S. Jin, and X. Xie, “Testing your question answering software via asking recursively,” in2021 36th IEEE/ACM International Confer- ence on Automated Software Engineering (ASE). IEEE, 2021, pp. 104– 116

    2021

  6. [16]

    Metal: Metamorphic testing framework for analyzing large-language model qualities,

    S. Hyun, M. Guo, and M. A. Babar, “Metal: Metamorphic testing framework for analyzing large-language model qualities,” in2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2024, pp. 117–128

    2024

  7. [17]

    Neural approaches to conversational ai,

    J. Gao, M. Galley, and L. Li, “Neural approaches to conversational ai,” inThe 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 1371–1374

    2018

  8. [18]

    Coqa: A conversational question answering challenge,

    S. Reddy, D. Chen, and C. D. Manning, “Coqa: A conversational question answering challenge,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 249–266, 2019

    2019

  9. [19]

    A survey of dialogue system evaluation,

    Y . Fan and X. Luo, “A survey of dialogue system evaluation,” in2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2020, pp. 1202–1209

    2020

  10. [20]

    Evaluating large language models: A comprehensive survey,

    Z. Guo, R. Jin, C. Liu, Y . Huang, D. Shi, L. Yu, Y . Liu, J. Li, B. Xiong, D. Xionget al., “Evaluating large language models: A comprehensive survey,”arXiv preprint arXiv:2310.19736, 2023

    2023 arXiv

  11. [21]

    Dialoglue: A natural language understanding benchmark for task-oriented dialogue,

    S. Mehri, M. Eric, and D. Hakkani-Tur, “Dialoglue: A natural language understanding benchmark for task-oriented dialogue,”arXiv preprint arXiv:2009.13570, 2020

    2009 arXiv

  12. [22]

    Drowzee: Metamorphic testing for fact-conflicting hallucination detection in large language models,

    N. Li, Y . Li, Y . Liu, L. Shi, K. Wang, and H. Wang, “Drowzee: Metamorphic testing for fact-conflicting hallucination detection in large language models,”Proc. ACM Program. Lang., vol. 8, no. OOPSLA2, Oct. 2024. [Online]. Available: https://doi.org/10.1145/3689776

    2024 doi

  13. [23]

    Leveraging large language models for nlg evaluation: Advances and challenges,

    Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu, Y . Lai, C. Tao, and S. Ma, “Leveraging large language models for nlg evaluation: Advances and challenges,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. C...

    2024

  14. [24]

    Dialogbench: Evaluating llms as human-like dialogue systems,

    J. Ou, J. Lu, C. Liu, Y . Tang, F. Zhang, D. Zhang, and K. Gai, “Dialogbench: Evaluating llms as human-like dialogue systems,”arXiv preprint arXiv:2311.01677, 2023. 16

    2023 arXiv

  15. [25]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models,”arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  16. [26]

    Parrot: Enhancing multi-turn instruction fol- lowing for large language models,

    Y . Sun, C. Liu, K. Zhou, J. Huang, R. Song, W. X. Zhao, F. Zhang, D. Zhang, and K. Gai, “Parrot: Enhancing multi-turn instruction fol- lowing for large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long ...

    2024

  17. [27]

    Mt-eval: A multi-turn capabili- ties evaluation benchmark for large language models,

    W.-C. Kwan, X. Zeng, Y . Jiang, Y . Wang, L. Li, L. Shang, X. Jiang, Q. Liu, and K.-F. Wong, “Mt-eval: A multi-turn capabili- ties evaluation benchmark for large language models,”arXiv preprint arXiv:2401.16745, 2024

    2024 arXiv

  18. [28]

    Mint: Evaluating llms in multi-turn interaction with tools and language feedback,

    X. Wang, Z. Wang, J. Liu, Y . Chen, L. Yuan, H. Peng, and H. Ji, “Mint: Evaluating llms in multi-turn interaction with tools and language feedback,”arXiv preprint arXiv:2309.10691, 2023

    2023 arXiv

  19. [29]

    BotChat: Evaluating LLMs’ capabilities of having multi-turn dialogues,

    H. Duan, J. Wei, C. Wang, H. Liu, Y . Fang, S. Zhang, D. Lin, and K. Chen, “BotChat: Evaluating LLMs’ capabilities of having multi-turn dialogues,” inFindings of the Association for Computational Linguistics: NAACL 2024, K. Duh, H. Gomez, and S. Bethard, Eds. Mexico City, Mexi...

    2024

  20. [30]

    A survey on metamorphic testing,

    S. Segura, G. Fraser, A. B. Sanchez, and A. Ruiz-Cort ´es, “A survey on metamorphic testing,”IEEE Transactions on software engineering, vol. 42, no. 9, pp. 805–824, 2016

    2016

  21. [31]

    Metamorphic test- ing: a new approach for generating next test cases,

    T. Y . Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic test- ing: a new approach for generating next test cases,”arXiv preprint arXiv:2002.12543, 2020

    2002 arXiv

  22. [32]

    Metamorphic testing: A review of challenges and opportunities,

    T. Y . Chen, F.-C. Kuo, H. Liu, P.-L. Poon, D. Towey, T. Tse, and Z. Q. Zhou, “Metamorphic testing: A review of challenges and opportunities,” ACM Computing Surveys (CSUR), vol. 51, no. 1, pp. 1–27, 2018

    2018

  23. [33]

    Machine learning test- ing: Survey, landscapes and horizons,

    J. M. Zhang, M. Harman, L. Ma, and Y . Liu, “Machine learning test- ing: Survey, landscapes and horizons,”IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 1–36, 2020

    2020

  24. [34]

    Natural test generation for precise testing of question answering software,

    Q. Shen, J. Chen, J. M. Zhang, H. Wang, S. Liu, and M. Tian, “Natural test generation for precise testing of question answering software,” inProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12

    2022

  25. [35]

    Dialtest: automated testing for recurrent- neural-network-driven dialogue systems,

    Z. Liu, Y . Feng, and Z. Chen, “Dialtest: automated testing for recurrent- neural-network-driven dialogue systems,” inProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2021, pp. 115–126

    2021

  26. [36]

    A metamorphic testing approach for assessing question answering systems,

    K. Tu, M. Jiang, and Z. Ding, “A metamorphic testing approach for assessing question answering systems,”Mathematics, vol. 9, no. 7, p. 726, 2021

    2021

  27. [37]

    Filling conversation ellipsis for better social dialog understanding,

    X. Zhang, C. Li, D. Yu, S. Davidson, and Z. Yu, “Filling conversation ellipsis for better social dialog understanding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, 2020, pp. 9587–9595

    2020

  28. [38]

    Mitkov,The Oxford handbook of computational linguistics

    R. Mitkov,The Oxford handbook of computational linguistics. Oxford university press, 2022

    2022

  29. [39]

    Anaphora and coreference resolution: A review,

    R. Sukthanker, S. Poria, E. Cambria, and R. Thirunavukarasu, “Anaphora and coreference resolution: A review,”Information Fusion, vol. 59, pp. 139–162, 2020

    2020

  30. [40]

    From local to global: A graph rag approach to query- focused summarization,

    D. Edge, H. Trinh, N. Cheng, J. Bradley, A. Chao, A. Mody, S. Truitt, and J. Larson, “From local to global: A graph rag approach to query- focused summarization,”arXiv preprint arXiv:2404.16130, 2024

    2024 arXiv

  31. [41]

    Extract, define, canonicalize: An llm- based framework for knowledge graph construction,

    B. Zhang and H. Soh, “Extract, define, canonicalize: An llm- based framework for knowledge graph construction,”arXiv preprint arXiv:2404.03868, 2024

    2024 arXiv

  32. [42]

    Qwen2 technical report,

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huanget al., “Qwen2 technical report,”arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  33. [43]

    Mistral-7b-instruct-v0.3 model,

    Mistral, “Mistral-7b-instruct-v0.3 model,” 2024. [Online]. Available: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

    2024

  34. [44]

    Llama 3 model card,

    AI@Meta, “Llama 3 model card,” 2024. [Online]. Available: https://github.com/meta-llama/llama3/blob/main/MODEL CARD.md

    2024

  35. [45]

    Team, “Gemma,” 2024

    G. Team, “Gemma,” 2024. [Online]. Available: https://www.kaggle. com/m/3301

    2024

  36. [46]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language mod- els are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  37. [47]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  38. [48]

    Know what you don’t know: Unanswerable questions for squad,

    P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018, pp. 784–789

    2018

  39. [49]

    Mutual: A dataset for multi-turn dialogue reasoning,

    L. Cui, Y . Wu, S. Liu, Y . Zhang, and M. Zhou, “Mutual: A dataset for multi-turn dialogue reasoning,” inProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 1406–1416

    2020

  40. [50]

    An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,

    H. Huang, Y . Qu, J. Liu, M. Yang, and T. Zhao, “An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge models are task- specific classifiers,”arXiv preprint arXiv:2403.02839, 2024

    2024 arXiv

  41. [51]

    Opal: Ontology- aware pretrained language model for end-to-end task-oriented dialogue,

    Z. Chen, Y . Liu, L. Chen, S. Zhu, M. Wu, and K. Yu, “Opal: Ontology- aware pretrained language model for end-to-end task-oriented dialogue,” Transactions of the Association for Computational Linguistics, vol. 11, pp. 68–84, 2023

    2023

  42. [52]

    Zero-shot information extraction via chatting with chatgpt,

    X. Wei, X. Cui, N. Cheng, X. Wang, X. Zhang, S. Huang, P. Xie, J. Xu, Y . Chen, M. Zhanget al., “Zero-shot information extraction via chatting with chatgpt,”arXiv preprint arXiv:2302.10205, 2023

    2023 arXiv

  43. [53]

    Testing graph database systems via graph-aware metamorphic relations,

    Z. Zhuang, P. Li, P. Ma, W. Meng, and S. Wang, “Testing graph database systems via graph-aware metamorphic relations,”Proceedings of the VLDB Endowment, vol. 17, no. 4, pp. 836–848, 2023

    2023

  44. [54]

    Metamorphic testing for software quality assessment: A study of search engines,

    Z. Q. Zhou, S. Xiang, and T. Y . Chen, “Metamorphic testing for software quality assessment: A study of search engines,”IEEE Transactions on Software Engineering, vol. 42, no. 3, pp. 264–284, 2015

    2015

  45. [55]

    Testing and validating machine learning classifiers by metamorphic testing,

    X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y . Chen, “Testing and validating machine learning classifiers by metamorphic testing,” Journal of Systems and Software, vol. 84, no. 4, pp. 544–558, 2011

    2011

  46. [56]

    Mttm: Metamorphic testing for textual content modera- tion software,

    W. Wang, J.-t. Huang, W. Wu, J. Zhang, Y . Huang, S. Li, P. He, and M. R. Lyu, “Mttm: Metamorphic testing for textual content modera- tion software,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2387–2399

    2023

  47. [57]

    Ontology-based metamorphic testing for chatbots,

    J. Bo ˇzi´c, “Ontology-based metamorphic testing for chatbots,”Software Quality Journal, vol. 30, no. 1, pp. 227–251, 2022

    2022

  48. [58]

    Knowledge graph driven inference testing for question answering software,

    J. Wang, Y . Li, Z. Chen, L. Chen, X. Zhang, and Y . Zhou, “Knowledge graph driven inference testing for question answering software,” inPro- ceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.