Pith. sign in

REVIEW 4 major objections 4 minor 68 references

The paper claims deep multi-turn understanding splits into six measurable capabilities, and shows the strongest of 22 model configurations satisfies all requirements in only 41.1% of responses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:49 UTC pith:4QGXVIZ2

load-bearing objection Genuinely useful six-mode multi-turn benchmark with careful construction, but the per-dimension leaderboard conflates the target capability with the model's own generation quality — a confound the paper itself demonstrates in F.2. the 4 major comments →

arxiv 2607.29196 v1 pith:4QGXVIZ2 submitted 2026-07-31 cs.CL

Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understanding

Eileen Ye , Jiawen Tao , Yaoming Li , Chenxu Liu , Wenhan Yu , Yaxin Fan , Xiaokun Yuan , Mengzhou Wu
show 2 more authors
Yanbing Jiang Maxm Pan
This is my paper
classification cs.CL
keywords multi-turn dialogueLLM benchmarkdialogue state trackingconstraint memoryaction suppressionreference resolutionChinese evaluationitem-level scoring
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Hy-MultiTurn argues that doing well in long, loosely structured multi-turn dialogue is not one general ability but at least six separable ones: remembering low-salience constraints, executing dense revisions, synthesizing the latest state, localizing objects under compound conditions, suppressing action until formal triggers are met, and resolving distant referents. To back this, the paper reverse-engineers the six mechanisms from real chatbot failures, rebuilds 209 controlled Chinese dialogues of 12–76 turns with traceable ground truths, and fixes the scoring before injecting 5–40+ irrelevant turns that cannot change the answer. Evaluation across 22 frontier configurations shows a large gap between partial credit and strict correctness: the best model reaches 77.2% weighted importance but only 41.1% strict accuracy, and different models lead different modes. The intended consequence is that holistic quality scores hide where models fail, and multi-turn competence should be measured and improved dimension by dimension rather than as one number.

Core claim

Hy-MultiTurn's central claim is that a correct final response in a long multi-turn conversation requires maintaining a unique valid dialogue state across six distinct mechanisms of failure: constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution. The paper operationalizes each mechanism as a controlled evaluation mode with a unique derivable answer, and scores responses twice: weighted importance for partial credit and strict accuracy for verbatim usability. The empirical discovery is that these two views diverge sharply—the strongest configuration is 77.2% by importance but 41.1% strict, and no configuration leads all six

What carries the argument

The carrying mechanism is the six-mode failure taxonomy plus a six-stage construction pipeline: anchor a mechanism to a real failure pattern, build a traceable ground truth (authoritative external sources for Mode I; a self-contained constraint graph, update chain, object registry, trigger boundary, or referent record for the others), render relevant turns and a final evaluation turn, derive and freeze item-level scoring, inject 5–40+ irrelevant turns that cannot change the frozen answer, and blind-select tasks from candidate pools. Scoring combines weighted rubric items, binary gate items that zero a response on catastrophic conclusions, deterministic checkers for Mode II, and strict accura

Load-bearing premise

The load-bearing premise of the main leaderboard is that letting each model generate its own earlier dialogue turns in Modes I, II, III, and V isolates the target mechanism; Section F.2 shows this premise is shaky, since removing or fixing assistant history changes scores by tens of points in some modes and models, mixing state-tracking ability with generation skill.

What would settle it

Recompute the leaderboard with fixed, user-only histories for all six modes. If Modes II and IV shift as much as the largest shift reported in Section F.2 (50.1 to 73.9), then the six dimensions are not isolated capability measures and the paper's per-mode diagnostics would need re-framing as diagnostics of the whole self-generation loop.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Professional use of multi-turn assistants requires verbatim correctness, and on that bar every evaluated configuration fails most of the time; the best passes only 41.1% of responses with all requirements met.
  • Models with similar aggregate scores fail differently, so selecting a model by a single leaderboard number is unreliable; per-dimension profiles are needed.
  • Action suppression is the least responsive to thinking mode and has the narrowest score spread, implying that withholding behavior needs alignment or control changes rather than longer reasoning.
  • Combining per-dimension leaders would reach 81.1% importance versus 77.2% for the best single model, so complementary model profiles are practically usable.
  • The benchmark is Chinese-native, and the paper's own limitation note says rankings may not transfer to English; cross-lingual validation is an open requirement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • My inference: Section F.2's fixed-context comparison covers only Modes I, III, and V; running the same comparison on Modes II and IV would tell whether the full per-dimension leaderboard is deconfounded from self-generated history quality.
  • My inference: the strict-versus-importance gap is probably more punishing in interactive agents, where a stale value or premature action can trigger tool calls; the paper raises this risk but does not test it.
  • My inference: the freeze-the-answer-before-distractors protocol is reusable as an audit for other benchmarks, because it separates mechanism-driven difficulty from generic length or noise difficulty.
  • My inference: because only 47.5% of the held-out substantive failures matched the six mechanisms, Hy-MultiTurn measures a diagnostic slice—state-control failures—rather than the full space of chatbot errors such as hallucination or tool failures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces Hy-MultiTurn, a Chinese benchmark for evaluating 'deep multi-turn dialogue understanding' across six dimensions: constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution. The six modes are induced from desensitized production chatbot failures and implemented as 209 controlled tasks spanning 12–76 turns, with a six-stage construction pipeline that anchors each task to a traceable ground truth, freezes answers and scoring before distractor injection, and blind-selects tasks for difficulty and discrimination. The evaluation of 22 frontier model configurations uses item-level weighted importance and strict accuracy, with three-roll averaging, LLM-as-judge scoring (except deterministic Mode II), and extensive reliability checks. The headline result is that even the best model, GPT-5.5, achieves only 41.1% strict accuracy, and different models lead on different dimensions.

Significance. If the central claims hold, Hy-MultiTurn is a valuable diagnostic benchmark: it is carefully constructed, with ground truth anchored to external sources or self-contained state, scoring frozen before distractors are added, a held-out coverage audit (72.2% of problematic conversations contain a substantive problem, 47.5% of those covered by the six mechanisms), high inter-judge agreement (κ=0.835), near-invariant judge rankings (Spearman ρ≥0.98), and evidence against family self-preference (+0.4pp, p=0.24). The item-level scoring and separate gate items for Modes I, V, and VI are thoughtful design choices that move beyond holistic quality scores. However, the main per-dimension leaderboard is confounded by self-generated dialogue history, and this threatens the paper's claim that the six capabilities are separately measurable. The paper's own auxiliary fixed-context analysis (Appendix F.2) shows the confound is large and direction-dependent, so the current presentation substantially overstates what the free-protocol results can establish.

major comments (4)
  1. [§4.1, §F.2, Table 2] The free-context protocol lets each model generate its own earlier assistant turns in Modes I, II, III, and V. Appendix F.2 quantifies the consequence: Mode III mean importance moves from 61.3 (free) to 75.1 (user-only), Mode I moves from 44.3 to 41.2, and the direction varies by model. For example, Gemini 3.1 Pro's Mode III score rises from 50.1 to 73.9, which would materially reorder Table 2 (where it appears at 53.8). Thus the per-dimension leaderboard conflates the target state-tracking capability with the model's generation quality, and the 'different models lead across the six modes' claim (§4.4) is not established for four of the six modes. The limitation paragraph (§6) mentions only cost growth, not this construct-validity threat.
  2. [§5.1, Fig. 7] The cross-dimension construct-validity correlations are computed on the same free-context importance scores. Since generation quality is correlated across dimensions and across models, the positive but non-uniform Spearman correlations (0.47–0.90) do not, by themselves, show that the six dimensions measure distinct capabilities. The conclusion that 'the six dimensions are not rescaled versions of one task' would be far more convincing if the correlation analysis were repeated on the fixed-context scores (user-only or full fixed) reported in §F.2, or if the authors showed that the pattern is robust to context source.
  3. [§3.4, §4.1] The difficulty controls—dialogue length and irrelevant-topic distraction—are intended to be varied independently (§3.4), but in the free protocol the effective history length and distractor content differ per model because each model generates its own earlier turns. This makes cross-model comparisons of per-dimension difficulty, and of the effects of the three generic difficulty factors, less controlled. Reporting the main per-dimension results on the fixed-context conditions, or at least alongside them, would restore the intended control and give a fairer test of each capability.
  4. [§3.3, Eq. (1)] Importance is the primary ranking metric, defined as Σ_{i∈H} w_i / Σ_i w_i, but the paper does not describe how the rubric weights w_i are assigned. If weights are arbitrary or author-chosen, the overall leaderboard and per-dimension rankings could be sensitive to that choice. Please specify the weighting procedure (e.g., equal weights, expert importance ratings, or derived from the ground-truth representation) and provide a sensitivity check, such as equal weights or a bounded weight perturbation, to show the main conclusions are stable.
minor comments (4)
  1. [§6] The Limitations section should explicitly acknowledge the construct-validity threat demonstrated in §F.2, not only the cost growth of self-generated histories. A sentence noting that per-dimension free-protocol scores mix generation quality with the target capability, and directing readers to the fixed-context analysis, would be appropriate.
  2. [Table 1] Hy-MultiTurn is marked ✓ for 'Objective/verifiable scoring', but five of six modes rely on LLM judges. Given the thorough reliability checks this is defensible, but a ✓/∼ mixed marker or a footnote clarifying that objectivity comes from anchored rubrics and judge averaging would be more precise.
  3. [General] The paper does not state a release URL, license, or data-availability plan for the benchmark, scoring rubrics, or judge prompts. For a benchmark paper, this information should be included; it is essential for reproducibility and community adoption.
  4. [§F.2, Table 8] The table uses ‡ markers and an explanation that two implementations are treated as equivalent. The caption is dense; making the two implementation variants a separate column or clearer footnote would improve readability and reduce the risk of misinterpretation.

Circularity Check

0 steps flagged

No significant circularity: Hy-MultiTurn is an empirical benchmark whose construction pipeline is explicitly designed to separate ground truth, scoring, and distractor injection, and whose results are measured rather than derived from fitted inputs.

full rationale

This is a benchmark-construction paper, not a derivation of predictions from fitted parameters. The six modes are induced from real chatbot failures, but the benchmark dialogues are regenerated under controlled conditions with no real user content, and the ground truth, answers, and scoring items are frozen before distractors are injected (Section 3.4, S2–S5). The coverage audit uses a separate held-out sample 'not used to define the six mechanisms, reducing circularity between mechanism induction and coverage estimation' (Supplementary Section C), which directly mitigates the main potential circularity in a failure-taxonomy benchmark. LLM-as-judge scoring is tested for same-family self-preference with a permutation test and found nonsignificant (Section 4.2). The only self-citation to Tencent's own Hy3 model repository is a model-card reference and is not load-bearing; Hy3 is not claimed to lead. The free-context protocol confound documented in Section F.2 is a real construct-validity concern for interpreting per-dimension rankings, but it is not a circularity: the per-dimension scores are empirical outcomes, not quantities that equal their own construction inputs by equation or by definition. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no result reduces to an ansatz smuggled in via citation. Accordingly, the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The benchmark introduces no new physical or mathematical entities; its load-bearing assumptions are about measurement validity: LLM judges, audit representativeness, the free-history protocol, and the generality of the six mechanisms. The free parameters are design choices (rubric weights, selection cutoffs, judge panel) that affect the exact numbers but are disclosed in spirit, not in full detail.

free parameters (3)
  • Rubric item weights w_i
    The importance metric sums weighted items, but the actual weights are not published; they are chosen by the authors and directly affect all importance scores and rankings.
  • Per-mode task selection cutoffs
    Candidates are filtered to final counts (e.g., Mode I 64→34, Mode V 341→35) based on cross-model difficulty and discrimination; these hand-chosen thresholds shape benchmark composition and difficulty.
  • Judge panel composition (GPT-5.2, Qwen 3.6, DeepSeek-V4 Pro)
    The three-judge mean is the basis of all reported scores; absolute scores depend on this choice, though the paper argues rankings are stable (Spearman ≥0.98).
axioms (4)
  • domain assumption LLM judges provide valid item-level correctness judgments in the five judge-scored modes.
    The entire leaderboard and per-dimension scores rest on the assumption that GPT-5.2, Qwen 3.6, and DeepSeek-V4 Pro correctly judge whether requirements are met. The paper reports inter-judge agreement but no human baseline.
  • domain assumption The held-out audit of 1,875 labeled conversations is representative of real chatbot failure distribution.
    The coverage estimate (47.5% of substantive problems match the six mechanisms) comes from a pre-filtered pool of likely problematic dialogues, not random traffic, and is labeled by an LLM-assisted rubric.
  • domain assumption Allowing models to generate their own dialogue histories in Modes I, II, III, and V does not invalidate cross-model comparisons of the target capability.
    Section F.2 shows that context source (free, full fixed, user-only) materially changes scores in some modes, e.g., Gemini 3.1 Pro Mode III: 50.1 free vs 73.9 user-only, indicating a confound between history generation and the measured capability.
  • domain assumption The six mechanisms induced from one large-scale Chinese chatbot generalize to the broader space of multi-turn dialogue failures.
    The taxonomy is grounded in a single production system; the coverage audit validates it on one held-out sample, but there is no external validation on other chatbots or languages.

pith-pipeline@v1.3.0-daily-deepseek · 30756 in / 12655 out tokens · 139164 ms · 2026-08-03T11:49:37.251474+00:00 · methodology

0 comments
read the original abstract

Long-running multi-turn interactions with chatbots and agents are now common, and a correct response often depends on remembering earlier details, tracking later revisions, identifying intended objects or referents, and withholding action when required conditions are unmet. Existing multi-turn benchmarks typically cover short exchanges and do not fully evaluate these capabilities in long multi-turn interactions, particularly in Chinese, while offering limited insight into how and why models fail. To address these limitations, we analyze real chatbot failures to identify six recurring mechanisms and use them to define six controlled evaluation modes in Hy-MultiTurn, a Chinese benchmark for deep multi-turn dialogue understanding. The six modes evaluate constraint memory, precise execution, constraint synthesis, object localization, action suppression, and reference resolution. Across the six modes, we construct 209 controlled tasks spanning 12-76 turns, with dialogue length, irrelevant-topic distraction, and colloquial phrasing adding further difficulty. Evaluation of 22 frontier model configurations shows that Hy-MultiTurn is broadly challenging, as even GPT-5.5, the strongest overall configuration, satisfies all requirements in only 41.1 percent of responses and no model performs best in all six modes.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 6 canonical work pages

  1. [1]

    Alibaba Cloud Model Studio : Model list

    Alibaba Cloud . Alibaba Cloud Model Studio : Model list. https://help.aliyun.com/zh/model-studio/models, 2026. Accessed: 2026-07-21

    2026

  2. [2]

    Claude Platform : Models overview

    Anthropic . Claude Platform : Models overview. https://platform.claude.com/docs/en/about-claude/models/overview, 2026. Accessed: 2026-07-21

    2026

  3. [3]

    MT-Bench-101 : A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. MT-Bench-101 : A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In ACL, 2024 a . doi:10.18653/v1/2024.acl-long.401. URL https://aclanthology.org/2024.acl-long.401/

    doi:10.18653/v1/2024.acl-long.401 2024

  4. [6]

    Doubao Seed Models : Model documentation

    ByteDance Seed . Doubao Seed Models : Model documentation. https://docs.volcengine.com/docs/82379?lang=zh, 2026. Accessed: 2026-07-29

    2026

  5. [7]

    DeepSeek API Docs : Models and pricing

    DeepSeek . DeepSeek API Docs : Models and pricing. https://api-docs.deepseek.com/quick_start/pricing/, 2026. Accessed: 2026-07-21

    2026

  6. [10]

    Hashimoto

    Yann Dubois, Balazs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval : A simple way to debias automatic evaluators. In COLM, 2024. URL https://arxiv.org/abs/2404.04475

    Pith/arXiv arXiv 2024

  7. [11]

    FairMT-Bench : Benchmarking fairness for multi-turn dialogue in conversational LLMs

    Zhiting Fan, Ruizhe Chen, Tianxiang Hu, and Zuozhu Liu. FairMT-Bench : Benchmarking fairness for multi-turn dialogue in conversational LLMs . In ICLR, 2025. URL https://openreview.net/forum?id=RSGoXnS9GH

    2025

  8. [12]

    Gemini API : Models

    Google . Gemini API : Models. https://ai.google.dev/gemini-api/docs/models, 2026. Accessed: 2026-07-21

    2026

  9. [16]

    RULER : What's the real context size of your long-context language models? In COLM, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER : What's the real context size of your long-context language models? In COLM, 2024. URL https://openreview.net/forum?id=kIoBbc76Sy

    2024

  10. [18]

    FollowBench : A multi-level fine-grained constraints following benchmark

    Yuxin Jiang et al. FollowBench : A multi-level fine-grained constraints following benchmark. In ACL, 2024. doi:10.18653/v1/2024.acl-long.257. URL https://aclanthology.org/2024.acl-long.257/

    doi:10.18653/v1/2024.acl-long.257 2024

  11. [19]

    BABILong : Testing the limits of LLMs with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong : Testing the limits of LLMs with long context reasoning-in-a-haystack. In NeurIPS, 2024. doi:10.52202/079017-3381. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c0d62e70dbc659cc9bd44cbcf1cb652f-Abstract-Datasets_and_Bench...

    work page doi:10.52202/079017-3381 2024

  12. [20]

    MT-Eval : A multi-turn capabilities evaluation benchmark for large language models

    Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. MT-Eval : A multi-turn capabilities evaluation benchmark for large language models. In EMNLP, 2024. doi:10.18653/v1/2024.emnlp-main.1124. URL https://aclanthology.org/2024.emnlp-main.1124/

    doi:10.18653/v1/2024.emnlp-main.1124 2024

  13. [21]

    LLMs get lost in multi-turn conversation

    Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multi-turn conversation. In ICLR, 2026. URL https://iclr.cc/virtual/2026/oral/10009147

    arXiv 2026

  14. [23]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. In ICML, pages 34209--34231, 2025 b . URL https://proceedings.mlr.press/v267/li25h.html

    2025

  15. [26]

    WildBench : Benchmarking LLMs with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench : Benchmarking LLMs with challenging tasks from real users in the wild. In ICLR, 2025. URL https://openreview.net/forum?id=MKEHCx25xp

    2025

  16. [27]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL, 12: 0 157--173, 2024 a . doi:10.1162/tacl_a_00638. URL https://aclanthology.org/2024.tacl-1.9/

    doi:10.1162/tacl_a_00638 2024

  17. [29]

    AgentBench : Evaluating LLMs as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. AgentBench : Evaluating LLMs as agents. In ICLR, 2024 c . URL https://openreview.net/forum?id=zAdUB0aCTQ

    2024

  18. [30]

    Chengqian Ma, Wei Tao, and Steven Y. Guo. C3 : A bilingual benchmark for spoken dialogue models exploring challenges in complex conversations. In EMNLP, 2025. doi:10.18653/v1/2025.emnlp-main.1160. URL https://aclanthology.org/2025.emnlp-main.1160/

    work page doi:10.18653/v1/2025.emnlp-main.1160 2025

  19. [31]

    Kimi API Platform : Model list

    Moonshot AI . Kimi API Platform : Model list. https://platform.kimi.ai/docs/models, 2026. Accessed: 2026-07-21

    2026

  20. [32]

    OpenAI Platform : Models

    OpenAI . OpenAI Platform : Models. https://platform.openai.com/docs/models, 2026. Accessed: 2026-07-21

    2026

  21. [36]

    Hy3 and Hy3 Preview : Model repositories

    Tencent Hunyuan . Hy3 and Hy3 Preview : Model repositories. https://github.com/Tencent-Hunyuan, 2026. Accessed: 2026-07-29

    2026

  22. [37]

    CMT-Eval : A novel chinese multi-turn dialogue evaluation dataset addressing real-world conversational challenges

    Siyu Tian, Kaijie Mo, Yupei Wang, and Renfen Hu. CMT-Eval : A novel chinese multi-turn dialogue evaluation dataset addressing real-world conversational challenges. In EMNLP Findings, 2025. doi:10.18653/v1/2025.findings-emnlp.992. URL https://aclanthology.org/2025.findings-emnlp.992/

    work page doi:10.18653/v1/2025.findings-emnlp.992 2025

  23. [38]

    MINT : Evaluating LLMs in multi-turn interaction with tools and language feedback

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. MINT : Evaluating LLMs in multi-turn interaction with tools and language feedback. In ICLR, 2024. URL https://openreview.net/forum?id=6xkjwqMs2E

    2024

  24. [39]

    Benchmarking complex instruction-following with multiple constraints composition

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction-following with multiple constraints composition. In NeurIPS, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/f8c24b08b96a08ec7a7a975fe...

    2024

  25. [40]

    LongMemEval : Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval : Benchmarking chat assistants on long-term interactive memory. In ICLR, 2025. URL https://iclr.cc/virtual/2025/poster/28290

    2025

  26. [41]

    xAI API : Models (grok 4.5)

    xAI . xAI API : Models (grok 4.5). https://docs.x.ai/docs/models, 2026. Accessed: 2026-07-21

    2026

  27. [42]

    Evaluating large language models at evaluating instruction following

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In ICLR, 2024. URL https://openreview.net/forum?id=tr0KidwPLc

    2024

  28. [44]

    IHEval : Evaluating language models on following the instruction hierarchy

    Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, et al. IHEval : Evaluating language models on following the instruction hierarchy. In NAACL, 2025. doi:10.18653/v1/2025.naacl-long.425. URL https://aclanthology.org/2025.naacl-long.425/

    doi:10.18653/v1/2025.naacl-long.425 2025

  29. [45]

    Judging LLM -as-a-judge with MT-Bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM -as-a-judge with MT-Bench and chatbot arena. In NeurIPS, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html

    2023

  30. [47]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou et al. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    Pith/arXiv arXiv 2023

  31. [48]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others , booktitle=. Judging. 2023 , url=

    2023

  32. [49]

    2024 , doi=

    Bai, Ge and Liu, Jie and Bu, Xingyuan and He, Yancheng and Liu, Jiaheng and Zhou, Zhanhui and Lin, Zhuoran and Su, Wenbo and Ge, Tiezheng and Zheng, Bo and others , booktitle=. 2024 , doi=

    2024

  33. [50]

    2024 , doi=

    Kwan, Wai-Chung and Zeng, Xingshan and Jiang, Yuxin and Wang, Yufei and Li, Liangyou and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wong, Kam-Fai , booktitle=. 2024 , doi=

    2024

  34. [51]

    He, Yun and Jin, Di and Wang, Chaoqi and Bi, Chloe and Mandyam, Karishma and Zhang, Hejia and Zhu, Chen and Li, Ning and Xu, Tengyu and Lv, Hongjiang and others , year=. Multi-. 2410.15553 , archivePrefix=

    Pith/arXiv arXiv

  35. [52]

    2026 , url=

    Laban, Philippe and Hayashi, Hiroaki and Zhou, Yingbo and Neville, Jennifer , booktitle=. 2026 , url=

    2026

  36. [53]

    2024 , url=

    Hsieh, Cheng-Ping and Sun, Simeng and Kriman, Samuel and Acharya, Shantanu and Rekesh, Dima and Jia, Fei and Ginsburg, Boris , booktitle=. 2024 , url=

    2024

  37. [54]

    2024 , doi=

    Kuratov, Yuri and Bulatov, Aydar and Anokhin, Petr and Rodkin, Ivan and Sorokin, Dmitry and Sorokin, Artyom and Burtsev, Mikhail , booktitle=. 2024 , doi=

    2024

  38. [55]

    , booktitle=

    Ma, Chengqian and Tao, Wei and Guo, Steven Y. , booktitle=. 2025 , doi=

    2025

  39. [56]

    2025 , doi=

    Tian, Siyu and Mo, Kaijie and Wang, Yupei and Hu, Renfen , booktitle=. 2025 , doi=

    2025

  40. [57]

    and Yue, Summer and Xing, Chen , booktitle=

    Deshpande, Kaustubh and Sirdeshmukh, Ved and Mols, Johannes Baptist and Jin, Lifeng and Hernandez-Cardona, Ed-Yeremai and Lee, Dean and Kritz, Jeremy and Primack, Willow E. and Yue, Summer and Xing, Chen , booktitle=. 2025 , publisher=. doi:10.18653/v1/2025.findings-acl.958 , url=

    doi:10.18653/v1/2025.findings-acl.958 2025

  41. [58]

    2025 , publisher=

    Li, Jinnan and Li, Jinzhe and Wang, Yue and Chang, Yi and Wu, Yuan , booktitle=. 2025 , publisher=. doi:10.18653/v1/2025.findings-acl.486 , url=

    doi:10.18653/v1/2025.findings-acl.486 2025

  42. [59]

    ACL Short Papers , year=

    Ellipsis-Dependent Reasoning: a New Challenge for Large Language Models , author=. ACL Short Papers , year=. doi:10.18653/v1/2023.acl-short.4 , url=

    work page doi:10.18653/v1/2023.acl-short.4 2023

  43. [60]

    TACL , volume=

    Lost in the Middle: How Language Models Use Long Contexts , author=. TACL , volume=. 2024 , doi=

    2024

  44. [61]

    2025 , doi=

    Zhang, Zhihan and Li, Shiyang and Zhang, Zixuan and Liu, Xin and Jiang, Haoming and Tang, Xianfeng and Gao, Yifan and Li, Zheng and Wang, Haodong and Tan, Zhaoxuan and others , booktitle=. 2025 , doi=

    2025

  45. [62]

    EMNLP Findings , year=

    Can Language Models Follow Multiple Turns of Entangled Instructions? , author=. EMNLP Findings , year=. doi:10.18653/v1/2025.findings-emnlp.1387 , url=

    doi:10.18653/v1/2025.findings-emnlp.1387 2025

  46. [63]

    2025 , url=

    Wu, Di and Wang, Hongwei and Yu, Wenhao and Zhang, Yuwei and Chang, Kai-Wei and Yu, Dong , booktitle=. 2025 , url=

    2025

  47. [64]

    One Battle After Another: Probing

    Jia, Qi and Shen, Ye and Song, Xiujie and Zhang, Kaiwei and Wang, Shibo and Pei, Dun and Zhu, Xiangyang and Zhai, Guangtao , booktitle=. One Battle After Another: Probing. 2026 , pages=. doi:10.18653/v1/2026.acl-long.433 , url=

    work page doi:10.18653/v1/2026.acl-long.433 2026

  48. [65]

    ACL , year=

    Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models , author=. ACL , year=. doi:10.18653/v1/2024.acl-long.525 , url=

    doi:10.18653/v1/2024.acl-long.525 2024

  49. [66]

    2023 , eprint=

    Instruction-Following Evaluation for Large Language Models , author=. 2023 , eprint=

    2023

  50. [67]

    2024 , doi=

    Jiang, Yuxin and others , booktitle=. 2024 , doi=

    2024

  51. [68]

    Play Favorites: A Statistical Method to Measure Self-Bias in

    Spiliopoulou, Evangelia and Fogliato, Riccardo and Burnsky, Hanna and Soliman, Tamer and Ma, Jie and Horwood, Graham and Ballesteros, Miguel , year=. Play Favorites: A Statistical Method to Measure Self-Bias in. 2508.06709 , archivePrefix=

    Pith/arXiv arXiv

  52. [69]

    2024 , pages=

    Bai, Yushi and Lv, Xin and Zhang, Jiajie and Lyu, Hongchang and Tang, Jiankai and Huang, Zhidian and Du, Zhengxiao and Liu, Xiao and Zeng, Aohan and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi , booktitle=. 2024 , pages=. doi:10.18653/v1/2024.acl-long.172 , url=

    doi:10.18653/v1/2024.acl-long.172 2024

  53. [70]

    2024 , pages=

    Zhang, Xinrong and Chen, Yingfa and Hu, Shengding and Xu, Zihang and Chen, Junhao and Hao, Moo and Han, Xu and Thai, Zhen and Wang, Shuo and Liu, Zhiyuan and Sun, Maosong , booktitle=. 2024 , pages=. doi:10.18653/v1/2024.acl-long.814 , url=

    doi:10.18653/v1/2024.acl-long.814 2024

  54. [71]

    2025 , pages=

    Li, Youquan and Zheng, Miao and Yang, Fan and Dong, Guosheng and Cui, Bin and Chen, Weipeng and Zhou, Zenan and Zhang, Wentao , booktitle=. 2025 , pages=. doi:10.18653/v1/2025.emnlp-main.471 , url=

    work page doi:10.18653/v1/2025.emnlp-main.471 2025

  55. [72]

    2024 , pages=

    Qin, Yiwei and Song, Kaiqiang and Hu, Yebowen and Yao, Wenlin and Cho, Sangwoo and Wang, Xiaoyang and Wu, Xuansheng and Liu, Fei and Liu, Pengfei and Yu, Dong , booktitle=. 2024 , pages=. doi:10.18653/v1/2024.findings-acl.772 , url=

    doi:10.18653/v1/2024.findings-acl.772 2024

  56. [73]

    NeurIPS , year=

    Benchmarking Complex Instruction-Following with Multiple Constraints Composition , author=. NeurIPS , year=

  57. [74]

    2024 , pages=

    Duan, Haodong and Wei, Jueqi and Wang, Chonghua and Liu, Hongwei and Fang, Yixiao and Zhang, Songyang and Lin, Dahua and Chen, Kai , booktitle=. 2024 , pages=. doi:10.18653/v1/2024.findings-naacl.201 , url=

    doi:10.18653/v1/2024.findings-naacl.201 2024

  58. [75]

    2024 , pages=

    Liu, Xiao and Lei, Xuanyu and Wang, Shengyuan and Huang, Yue and Feng, Zhuoer and Wen, Bosi and Cheng, Jiale and Ke, Pei and Xu, Yifan and Tam, Weng Lam and Zhang, Xiaohan and Sun, Lichao and Gu, Xiaotao and Wang, Hongning and Zhang, Jing and Huang, Minlie and Dong, Yuxiao and Tang, Jie , booktitle=. 2024 , pages=. doi:10.18653/v1/2024.acl-long.624 , url=

    doi:10.18653/v1/2024.acl-long.624 2024

  59. [76]

    2025 , url=

    Lin, Bill Yuchen and Deng, Yuntian and Chandu, Khyathi and Brahman, Faeze and Ravichander, Abhilasha and Pyatkin, Valentina and Dziri, Nouha and Le Bras, Ronan and Choi, Yejin , booktitle=. 2025 , url=

    2025

  60. [77]

    ICML , year=

    From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and BenchBuilder Pipeline , author=. ICML , year=

  61. [78]

    , booktitle=

    Dubois, Yann and Galambosi, Balazs and Liang, Percy and Hashimoto, Tatsunori B. , booktitle=. Length-Controlled. 2024 , url=

    2024

  62. [79]

    ICLR , year=

    Evaluating Large Language Models at Evaluating Instruction Following , author=. ICLR , year=

  63. [80]

    2024 , url=

    Wang, Xingyao and Wang, Zihan and Liu, Jiateng and Chen, Yangyi and Yuan, Lifan and Peng, Hao and Ji, Heng , booktitle=. 2024 , url=

    2024

  64. [81]

    2024 , url=

    Liu, Xiao and Yu, Hao and Zhang, Hanchen and Xu, Yifan and Lei, Xuanyu and Lai, Hanyu and Gu, Yu and Ding, Hangliang and Men, Kaiwen and Yang, Kejuan and others , booktitle=. 2024 , url=

    2024

  65. [82]

    2305.10250 , archivePrefix=

    Zhong, Wanjun and Guo, Lianghong and Gao, Qiqi and Ye, He and Wang, Yanlin , year=. 2305.10250 , archivePrefix=

    Pith/arXiv arXiv

  66. [83]

    2405.19444 , archivePrefix=

    Liang, Zhenwen and Yu, Dian and Yu, Wenhao and Yao, Wenlin and Zhang, Zhihan and Zhang, Xiangliang and Yu, Dong , year=. 2405.19444 , archivePrefix=

    Pith/arXiv arXiv

  67. [84]

    2412.15204 , archivePrefix=

    Bai, Yushi and Tu, Shangqing and Zhang, Jiajie and Peng, Hao and Wang, Xiaozhi and Lv, Xin and Cao, Shulin and Xu, Jiazheng and Hou, Lei and Dong, Yuxiao and Tang, Jie and Li, Juanzi , year=. 2412.15204 , archivePrefix=

    Pith/arXiv arXiv

  68. [85]

    2025 , url=

    Fan, Zhiting and Chen, Ruizhe and Hu, Tianxiang and Liu, Zuozhu , booktitle=. 2025 , url=

    2025