Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework

Fan Lin; Linlin Song; Qingsong Lv; Shupei Xiao; Shuyi Xie; Sui Wenjuan; Zishan Xu

REVIEW 4 major objections 5 minor 53 references

Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fine-tuning a 7B open-source model on a new error-attribution dataset yields a judge that scores, categorizes, and explains LLM responses better than GPT-4.

desk verdict AttriData and MisAttributionLLM are a real applied contribution, but the broad 'general-purpose' attribution claim needs out-of-domain evidence before it will stand. read the letter →

arxiv 2507.08459 v1 pith:N4LTFVKN submitted 2025-07-11 cs.CL

Zishan Xu , Shuyi Xie , Qingsong Lv , Shupei Xiao , Linlin Song , Sui Wenjuan , Fan Lin This is my paper

classification cs.CL

keywords LLMevaluationerrorattributionjudgemodelMisAttriDatamisattributionframeworkfeedbackgenerationhumanalignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single open-source judge model can do more than grade an LLM answer: it can also say what kind of error the answer contains and explain why. The authors build a 15-category Misattribution Framework, annotate 21,702 real-user samples with score, error label, and feedback, and fine-tune a 7B model on that data. On the test set the model reaches 0.935 Pearson correlation with human scores, 0.829 micro-F1 on error classification, and is preferred over GPT-4 in 60.41% of pairwise feedback comparisons. If the results hold, a 7B open-source judge can rival or beat large closed models at scoring, explaining, and categorizing LLM failures.

What carries the argument

The load-bearing mechanism is the combination of a fixed taxonomy and supervised fine-tuning. The Misattribution Framework organizes errors into 6 primary categories (Instruction Following, Response Quality, Knowledge Ability, Reasoning Capability, Safety, Other Errors) and 15 secondary labels. AttriData maps each sample to one score (0-3), one most-relevant error label, and one feedback text; fine-tuning Qwen2.5-7B on 18,806 training samples teaches the model to emit all three in a fixed order. The taxonomy is what turns open-ended critique into a structured diagnostic.

What would settle it

Take a held-out set of English-only questions from an independent source, run MisAttributionLLM zero-shot, and measure how often its error label matches human annotation; if micro-F1 falls near GPT-4's level or a large share of labels land in 'Other Errors', the framework's generality is not supported.

Watch

Extended reading notes

Core claim

The central claim is that error attribution—naming the most relevant failure category—is a learnable and useful component of LLM evaluation, and that a 7B model fine-tuned on AttriData can provide it jointly with a score and feedback. The authors report that MisAttributionLLM outperforms GPT-4 on the AttriData test set in correlation with human scoring (0.935 vs. 0.817 Pearson), in multi-class error attribution (0.829 vs. 0.731 micro-F1), and in human preference for its feedback (60.41% win rate). They also report that removing misattribution labels from training degrades error detection sharply, evidence that attribution training is what carries the capability.

Load-bearing premise

The load-bearing premise is that the 15-category Misattribution Framework is complete and general enough to cover the errors that actually occur in real LLM use; the paper itself concedes the framework does not cover all error types and relies on an 'Other Errors' catch-all for the rest.

Editorial extensions

If this is right

LLM platforms can automatically sort failures into categories such as hallucination, process error, refusal, or safety, instead of reviewing each response manually.
An open-source 7B judge can deliver scoring and feedback without the cost and data-leakage risks of commercial APIs.
The ablation result implies that error-attribution training materially improves error detection: without misattribution labels, precision falls from 0.985 to 0.348 and F1 from 0.970 to 0.516.
Error attribution turns evaluation output into targeted signals for pre-training and post-training fixes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the question set is drawn from a Chinese real-world evaluation source and only about 6% of AttriData is English, the claimed generality should be read as demonstrated mainly for Chinese-language user interactions.
With only 8 annotated Safety samples, the framework's coverage of safety failures is asserted rather than measured.
A direct test of generality would be zero-shot application to an English benchmark from a different platform; the published AlignBench result (Pearson 0.779 vs. GPT-4's 0.786) is still Chinese and uses the same prompt scheme.
The single-label design cannot represent responses with several simultaneous error types, so its diagnostic output is a simplification for multi-error answers.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a better than average applied LLM-evaluation paper, and the central combination is new: a single 7B judge that scores, assigns a misattribution category, and writes feedback. The AttriData construction is the most valuable part. The annotation process is serious—36 annotators plus 12 senior experts, three independent annotations per sample, senior review, and Fleiss kappa of 0.875 for score and 0.832 for misattribution. Those are respectable numbers for this kind of data. The ablation is also convincing: dropping misattribution annotations from training makes precision collapse from 0.985 to 0.348, which shows the attribution signal is real and not just a re-labeling artifact. The reported in-domain results—Pearson 0.935, micro-F1 0.829, 60.41% pairwise win over GPT-4 on feedback—are internally consistent with a well-tuned fine-tuned judge.

I agree with the stress-test note: the general-purpose claim is the weak point. All misattribution numbers come from the AttriData test split, which is a random split from the same Chinese, TencentLLMEval-derived pool used for training. That makes the test set part of the same distribution, so the model is being evaluated where it was trained. The only external benchmark, AlignBench, measures score correlation, not attribution. On top of that, the taxonomy is badly imbalanced: Safety has 8 samples, Other Errors has 129, and Reasoning Capability has 4,839 out of 8,026. Much of the micro-F1 could be driven by one or two frequent categories. The Chinese skew and the closed-source answer generation (ERNIE Bot and Hunyuan) make it harder to call the framework general. The paper's own limitation section concedes both the incomplete taxonomy and the GPT-4 ceiling on feedback, which is honest but does not fix the evidence gap.

Other soft spots are minor by comparison. The dataset itself is not released yet, only code; for a resource paper that is a problem. The three experimental runs are averaged without variance or intervals, so we cannot see how stable the 0.935 or 0.829 really are. These are fixable rather than fatal.

Who is this for? Anyone working on LLM-as-a-judge or automated error analysis will want to know about AttriData and the misattribution approach. As a general-purpose attribution tool, it is not yet demonstrated. As an in-domain judge plus a well-documented dataset and taxonomy, it is a legitimate contribution. I would not desk-reject it. Send it to peer review with the expectation of major revision: release or at least sample the data, add an out-of-distribution attribution evaluation with balanced categories, report per-category F1, include error bars, and soften the general-purpose wording.

Referee Report

4 major / 5 minor

Summary. The paper introduces a Misattribution Framework with 6 primary and 15 secondary error categories, constructs AttriData (21,702 samples, 8,026 with misattribution, mostly Chinese and derived from TencentLLMEval), and fine-tunes Qwen2.5-7B to produce feedback, misattribution, and a score in one pass. On the AttriData test split, the resulting model, MisAttributionLLM, is reported to achieve a Pearson correlation of 0.935 with human scores, a misattribution detection F1 of 0.970, a multi-class attribution micro-F1 of 0.829, and pairwise feedback win rates of 60.41% over GPT-4 and 85.20% over GPT-3.5. An additional AlignBench experiment reports scoring correlation only and does not evaluate error attribution.

Significance. If the results generalize, the paper would provide a useful open-source judge model and a reusable dataset for a relatively underexplored capability: error attribution in addition to scoring and feedback. The strengths include a human-annotated dataset with reported Fleiss kappa agreement, a broad set of baselines, an ablation showing the importance of misattribution training data, an external scoring benchmark, and a commitment to releasing code. However, the central claim that MisAttributionLLM is a general-purpose error-attribution model is not yet supported by the evidence: the attribution evaluation is in-domain, the taxonomy is highly imbalanced with very few Safety and Other Errors samples, and the external benchmark does not test attribution. The current evidence establishes a strong in-domain attribution model rather than a demonstrated general-purpose one.

major comments (4)

[§3.2, Tables 9–10; §4.3, Appendix C] The central claim that MisAttributionLLM provides general-purpose error attribution rests entirely on the AttriData test split, which is a random split of the same Chinese, TencentLLMEval-derived pool used for training. The AlignBench experiment in Appendix C measures only score correlation and does not evaluate misattribution, so it cannot validate the attribution capability out of distribution. Please add an out-of-distribution misattribution evaluation, for example on English data or on an independent error-analysis benchmark, and report attribution accuracy there; otherwise the headline claim should be explicitly scoped to the distribution represented by AttriData.
[§4.3, Table 10 and Table 4] The reported micro-F1 of 0.829 for multi-class error attribution is dominated by the Reasoning Capability category, which accounts for 4,839 of 8,026 misattribution instances, while Safety has only 8 instances and Other Errors has 129. Micro-averaged F1 can conceal poor performance on rare classes, so the current result does not demonstrate that the 15-category framework is reliable across categories. Please report per-category precision, recall, and F1, or at least macro-F1, and discuss performance on low-resource categories; this is necessary to support the framework's claimed comprehensiveness.
[§4.3, Figure 3, and Limitation] The feedback quality comparison is not a clean independent head-to-head: the feedback in AttriData was generated by GPT-4 and then human-corrected, so MisAttributionLLM was trained on targets derived from GPT-4. The Limitation section acknowledges that feedback quality is bounded by GPT-4, but the abstract and introduction still claim that MisAttributionLLM outperforms GPT-4 on feedback. To support this claim, please report inter-annotator agreement for the pairwise preference judgments, clarify how the 949 compared samples were selected, and ideally evaluate feedback on examples whose reference feedback was not produced by GPT-4.
[§3.1 and §4.3] The paper describes the taxonomy as 'comprehensive' while simultaneously using an 'Other Errors' catch-all category and acknowledging in the Limitation section that the framework does not cover all error types. This tension affects the interpretation of the 0.829 micro-F1: a high score could partly reflect the model's ability to predict the dominant Reasoning Capability class rather than to attribute a diverse set of errors. Please either provide evidence of category-level coverage (e.g., low proportion of Other Errors after annotation, with a breakdown of what falls into that category) or qualify the comprehensiveness claim.

minor comments (5)

[§4.3 and Figure 4] There are typos in the text: 'Appledix C' should be 'Appendix C', and 'Gound Truth' in Figure 4 should be 'Ground Truth'.
[Figure 3] Figure 3 appears in the manuscript as a string of '/uni000...' tokens rather than a readable chart; please replace it with the actual bar chart or a table of win/lose/tie percentages.
[§3.1] The sentence 'the detailed examples can be referenced in Appendix 8 and Appendix 9' refers to numbered sections that do not match the named appendices; these should be renamed to Figure 8 and Figure 9 or to the corresponding appendix letters.
[§4.3] The method is described as 'multi-classification' despite the annotation using a single 'most relevant' label per instance; please clarify in the text that this is single-label classification over 15 categories, not multi-label classification.
[Appendix C] The AlignBench correlation results are reported without confidence intervals or significance tests; adding these would make the comparison with GPT-4 more informative.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the model is trained and evaluated on a held-out split of an independently annotated dataset, the GPT-4 teacher comparison is a fair head-to-head, and self-citations serve only as data provenance.

full rationale

The paper's derivation chain is a standard supervised learning pipeline: define a taxonomy (Section 3.1), construct and manually annotate AttriData (Section 3.2), fine-tune Qwen2.5-7B on that data (Section 3.4), and evaluate on a held-out random split (Section 4). No equation or claim reduces to its own inputs by construction. The Pearson 0.935 score correlation and micro-F1 0.829 attribution results are measurements on the AttriData test set, which is an ordinary in-distribution held-out evaluation rather than a fitted-input prediction. The feedback comparison with GPT-4 is also not circular: although GPT-4 generated the initial feedback later human-verified for training, the pairwise test is an anonymous inference-time comparison, and a student model can legitimately surpass its teacher. The data sources Xie et al. (2023) and Lin et al. (2024) include current authors, but those citations only describe question collection and prompt-generation methodology; they are provenance, not load-bearing support for the central claim. The paper's own Limitation section concedes that the framework is incomplete and that 'Other Errors' is a placeholder, and the external AlignBench experiment tests only scoring, not attribution; these are generalization and scope limitations, not circularity. The claimed 'general-purpose' status is an extrapolation from one Chinese-skewed in-domain test set, but that is a correctness or evidence-strength concern, explicitly excluded from the circularity rubric.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the human labels, the taxonomy, the data source, and the GPT-4-derived feedback; none of these are derived in the paper, so they are axioms rather than results.

assumptions (5)

domain assumption Human-annotated scores and misattribution labels in AttriData are correct and consistent enough to serve as ground truth.
The paper uses human labels to fine-tune and evaluate the judge model; inter-annotator agreement is high (Fleiss kappa 0.875 and 0.832), but the categories are subjective, especially for borderline errors.
domain assumption Correlation with human scores is a valid proxy for judge quality.
The main evaluation follows prior judge-model work (e.g., CritiqueLLM) and uses Pearson, Spearman, and Kendall correlations against human scores; this assumes human scoring is the gold standard.
ad hoc to paper The error taxonomy is sufficiently comprehensive and non-redundant for general-purpose error attribution.
The 6 primary and 15 secondary categories were constructed by the authors from prior work and their own analysis; the paper acknowledges incompleteness and uses 'Other Errors' as a catch-all.
domain assumption Questions from TencentLLMEval, refined by IDGen, represent real-world user demands.
The question pool is derived from the authors' own prior datasets (Xie et al. 2023; Lin et al. 2024) and is predominantly Chinese; representativeness for global, multilingual platforms is not established.
domain assumption GPT-4-generated feedback, after human verification, is a suitable training target for feedback generation.
The paper states that feedback quality is constrained by GPT-4 (Limitation section), meaning the fine-tuned model cannot exceed the corrected-GPT-4 ceiling on feedback.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework." pith.science (2026). https://pith.science/paper/N4LTFVKN

@misc{pith2026250708459,
  author       = {Pith},
  title        = {Pith review of: Diagnosing Failures in Large Language Models' Answers: Integrating Error Attribution into Evaluation Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N4LTFVKN}},
  note         = {Machine review of arXiv:2507.08459}
}

read the original abstract

With the widespread application of Large Language Models (LLMs) in various tasks, the mainstream LLM platforms generate massive user-model interactions daily. In order to efficiently analyze the performance of models and diagnose failures in their answers, it is essential to develop an automated framework to systematically categorize and attribute errors. However, existing evaluation models lack error attribution capability. In this work, we establish a comprehensive Misattribution Framework with 6 primary and 15 secondary categories to facilitate in-depth analysis. Based on this framework, we present AttriData, a dataset specifically designed for error attribution, encompassing misattribution, along with the corresponding scores and feedback. We also propose MisAttributionLLM, a fine-tuned model on AttriData, which is the first general-purpose judge model capable of simultaneously generating score, misattribution, and feedback. Extensive experiments and analyses are conducted to confirm the effectiveness and robustness of our proposed method.

Figures

Figures reproduced from arXiv: 2507.08459 by the authors.

**Figure 2.** The overview of our method. The process can be generally divided into three main steps: data construction, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The results of pairwise comparison of the quality of the feedback generated by GPT-4, MisAttributionLLM [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Some examples of GPT-4, GPT-3.5 and MisAttributionLLM on AttriData test dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

**Figure 5.** Figure 5: The prompt template of feedback generation by GPT-4. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: The prompt utilized in the experiments in English. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: The prompt utilized in the experiments in Chinese. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: The case of Response Quality and Instruction Following. The wrong information was highlighted in red. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: The case of Knowledge Ability, Reasoning Capability and Safety. The wrong information was highlighted [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 20 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. In Proceedings of the AAAI Conference on Artificial Intelligence, 11, pages 12317--12325

work page 2022
[3]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

2024
[4]

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.01294

arXiv 2021
[5]

Doubao Team . 2024. Doubao pro models. https://team.doubao.com/en/. Accessed: 2024-09-25

work page 2024
[6]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

arXiv 2024
[7]

David Harbecke, Yuxuan Chen, Leonhard Hennig, and Christoph Alt. 2022. Why only micro-f1? class weighting of measures for relation classification. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 32--41

work page 2022
[8]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

2023

Show all 53 references

[9]

Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023. Tigerscore: Towards building explainable metric for all text generation tasks. Transactions on Machine Learning Research

2023
[10]

Ryo Kamoi, Sarkar Snigdha Sarathi Das, Renze Lou, Jihyun Janice Ahn, Yilun Zhao, Xiaoxin Lu, Nan Zhang, Yusen Zhang, Ranran Haoran Zhang, Sujeeth Reddy Vummanthala, et al. 2024 a . Evaluating llms at detecting errors in llm responses. arXiv preprint arXiv:2404.03602

2024 arXiv
[11]

Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024 b . When can llms actually correct their own mistakes? a critical survey of self-correction of llms. arXiv preprint arXiv:2406.01297

2024 arXiv
[12]

Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, et al. 2024. Critiquellm: Towards an informative critique generation model for evaluation of large language model generation. In Proceedings of the 62nd An...

2024
[13]

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

2023
[14]

Diederik P Kingma. 2015. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

2015 arXiv
[15]

Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332

2021 arXiv
[16]

JR Landis. 1977. The measurement of observer agreement for categorical data. Biometrics

1977
[17]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[18]

Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Hu Xu, Zishan, Xiao Zhichao, Yuhong Xiao, Liu, and Yu Zhang. 2024. Idgen: Item discrimination induced prompt generation for llm evaluation. Advances in Neural Information Processing Systems

2024
[19]

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. 2023 a . Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743

2023 arXiv
[20]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2024. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations

2024
[21]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634

2023 arXiv
[22]

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of t...

2023
[23]

Filip Moons and Ellen Vandervieren. 2023. Measuring agreement among several raters classifying subjects into one-or-more (hierarchical) nominal categories. a generalisation of fleiss' kappa. arXiv preprint arXiv:2303.12502

2023
[24]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...

2022
[25]

Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with frank: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin...

2021
[26]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[27]

Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan. 2023. Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models. arXiv preprint arXiv:2307.08487

2023 arXiv
[28]

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1--16. IEEE

2020
[29]

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...

2020
[30]

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

2020 arXiv
[31]

Zhisheng Tang, Ke Shen, and Mayank Kejriwal. 2024. An evaluation of estimative uncertainty in large language models. arXiv preprint arXiv:2405.15185

2024 arXiv
[32]

Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. 2023 a . Decodingtrust: a comprehensive assessment of trustworthiness in gpt models. In Proceedings of the 37th International Conference ...

2023
[33]

Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, et al. 2023 b . Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. In The Twelfth International Conference on Learning...

2023
[34]

Yaozu Wu, Dongyuan Li, Yankai Chen, Renhe Jiang, Henry Peng Zou, Liancheng Fang, Zhen Wang, and Philip S Yu. 2025. Multi-agent autonomous driving systems with large language models: A survey of recent advances. arXiv preprint arXiv:2502.16804

2025
[35]

Shuyi Xie, Wenlin Yao, Yong Dai, Shaobo Wang, Donlin Zhou, Lifeng Jin, Xinhua Feng, Pengzhi Wei, Yujie Lin, Zhichao Hu, et al. 2023. Tencentllmeval: a hierarchical evaluation of real-world capabilities for human-aligned llms. arXiv preprint arXiv:2311.05374

2023 arXiv
[36]

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305

2023 arXiv
[37]

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

2024 arXiv
[38]

Jingheng Ye, Zishan Xu, Yinghui Li, Xuxin Cheng, Linlin Song, Qingyu Zhou, Hai-Tao Zheng, Ying Shen, and Xin Su. 2024 a . Cleme2. 0: Towards more interpretable evaluation by disentangling edits for grammatical error correction. arXiv preprint arXiv:2407.00934

2024 arXiv
[39]

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2023. Flask: Fine-grained language model evaluation based on alignment skill sets. In ICLR 2024 Workshop on Large Language Model (LLM) Agents

2023
[40]

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024 b . Flask: Fine-grained language model evaluation based on alignment skill sets. In The Twelfth International Conference on Learning Representations

2024
[41]

Xunjian Yin and Xiaojun Wan. 2022. How do seq2seq models perform on end-to-end data-to-text generation? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7701--7710

2022
[42]

Dianzhi Yu, Xinni Zhang, Yankai Chen, Aiwei Liu, Yifei Zhang, Philip S Yu, and Irwin King. 2024. Recent advances of multimodal continual learning: A comprehensive survey. arXiv preprint arXiv:2410.05352

2024
[43]

Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263--27277

2021
[44]

Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations

2024
[45]

Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023 a . How language model hallucinations can snowball. arXiv preprint arXiv:2305.13534

2023 arXiv
[46]

Nan Zhang, Yusen Zhang, Wu Guo, Prasenjit Mitra, and Rui Zhang. 2023 b . Famesumm: Investigating and improving faithfulness of medical summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10915--10931

2023
[47]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

2019 arXiv
[48]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

2024
[49]

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024 b . Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372

2024 arXiv
[50]

Henry Peng Zou, Zhengyao Gu, Yue Zhou, Yankai Chen, Weizhi Zhang, Liancheng Fang, Yibo Wang, Yangning Li, Kay Liu, and Philip S Yu. 2025 a . Testnuc: Enhancing test-time computing approaches through neighboring unlabeled data consistency. arXiv preprint arXiv:2502.19163

2025 arXiv
[51]

Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Yankai Chen, Chunyu Miao, Hoang Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Langzhou He, et al. 2025 b . A survey on large language model based human-agent systems. arXiv preprint arXiv:2505.00753

2025 arXiv
[52]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[53]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. In Proceedings of the AAAI Conference on Artificial Intelligence, 11, pages 12317--12325

work page 2022

[3] [3]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

2024

[4] [4]

Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A Smith, and Yejin Choi. 2021. Is gpt-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. arXiv preprint arXiv:2107.01294

arXiv 2021

[5] [5]

Doubao Team . 2024. Doubao pro models. https://team.doubao.com/en/. Accessed: 2024-09-25

work page 2024

[6] [6]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

arXiv 2024

[7] [7]

David Harbecke, Yuxuan Chen, Leonhard Hennig, and Christoph Alt. 2022. Why only micro-f1? class weighting of measures for relation classification. In Proceedings of NLP Power! The First Workshop on Efficient Benchmarking in NLP, pages 32--41

work page 2022

[8] [8]

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

2023