Pith. sign in

REVIEW 4 major objections 7 minor 43 references

MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Markdown Awareness is measurable: MDEval matches humans on 84.1% of pairwise formatting comparisons and lifts weaker models by fine-tuning.

desk verdict A genuinely new benchmark for Markdown formatting, but the headline accuracy claim is cherry-picked (ties skipped) and needs an honest rewrite before the numbers can be trusted. read the letter →

arxiv 2501.15000 v2 pith:7MBVEOA5 submitted 2025-01-25 cs.CL cs.IR

Zhongpu Chen , Yinfeng Liu , Long Shi , Xingyan Chen , Yu Zhao , Fuji Ren This is my paper · ORCID
classification cs.CLcs.IR
keywords MarkdownAwarenessLLMevaluationbenchmarkwebchatbotreadabilityhumanalignmentLevenshteindistancestructure-orientedmetricsupervisedfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Markdown Awareness is the quality of how a large language model structures a response with Markdown — headings, lists, bold, code, and math — when the prompt does not ask for formatting. The paper argues this is an overlooked ability with direct impact on web-chatbot readability, and introduces MDEval, a 20,000-prompt benchmark across ten subjects in English and Chinese to measure it. MDEval scores each response by having a fixed judge model rewrite it into a well-structured reference, converting both versions into HTML, and computing the normalized edit distance between the extracted tag sequences. The authors report 84.1% agreement with human pairwise preference and a Spearman correlation of 0.791, and show that fine-tuning a 13B open-source model on the curated data raises its Markdown Awareness to a level the paper describes as comparable to GPT-4o.

What carries the argument

The HTMLification-and-edit-distance pipeline is the load-bearing mechanism: each response and its judge-written reference are rendered to HTML, the tags are extracted as a string, and the Markdown Awareness score is $$1 - \text{editDistance}(r, \hat{r})/\max(\text{len}(r), \text{len}(\hat{r})).$$ This recasts "how well structured is this answer" as "how few tag-operations separate it from a high-quality rewrite of the same content," giving an interpretable, content-free measure. A secondary mechanism is the Elo-rating crowdsourcing harness, which provides the human ground truth the metric is validated against.

What would settle it

Re-run the benchmark with a different judge LLM or a different rewrite prompt; if model rankings shift materially, the metric is judge-dependent. Or run a preference study in which humans choose between a model's original response and its GPT-4o rewrite: the paper's alignment claim predicts the rewrite should win most pairings, so frequent human choices for the original or a third style would falsify the claim that the edit-distance proxy captures human readability.

Watch

Extended reading notes

Core claim

Markdown Awareness, as defined here, is a structure-oriented, ground-truth-free score: for a given prompt and model, the model's response is rewritten on the fly by a leading LLM into an equivalent but well-structured Markdown version, and the two are compared purely as HTML tag strings. Custom <math> tags preserve LaTeX equations that standard Markdown parsers drop. The score is 1 minus the normalized Levenshtein distance, so 1 means the model's own formatting already matches the rewrite. The paper's central evidence is that this score tracks humans: in a pairwise Elo-based crowdsourcing study, MDEval reaches 84.1% record-level accuracy and 0.791 Spearman correlation, beating the LLM-judge baselines P-LLM and R-LLM and the rule-based D-Rule. The benchmark also doubles as a training set, with supervised fine-tuning lifting Baichuan2-13b-chat-v1 from 0.171 to about 0.73.

Load-bearing premise

The measurement assumes that GPT-4o's rewrite of a response, produced with a single fixed prompt, is the correct reference for well-structured Markdown, and that tag edit distance to that reference is what humans mean by readability; if either fails, the rankings may reward mimicry of one formatting style instead of general readability.

Editorial extensions

If this is right

  • Model rankings for chatbot readability can be obtained automatically, without human judges or content-specific ground truth, at the cost of one rewrite call and one string comparison per response.
  • The dataset doubles as a supervised fine-tuning resource, so weaker open models can be lifted toward the formatting quality of much stronger models.
  • Because each reference is a rewrite of the model's own words, the benchmark sidesteps the problem that no single canonical Markdown answer exists for a prompt.
  • Since scoring works on HTML tags, the same pipeline can be adapted to other structured output formats such as plain HTML, wiki markup, or table-heavy layouts.
  • The resulting rankings differ from general leaderboards: Deepseek-v2-chat beats GPT-4o, and Llama-3.1-8b beats Claude-3.5-sonnet, suggesting Markdown Awareness is a distinct capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If formatting taste is plural, a single judge model and prompt may bake in one style; swapping the judge or averaging several judges would reveal how much of the ranking is an artifact of GPT-4o's own formatting preferences.
  • The edit distance treats all tag edits equally, so a missing heading and a reordered list cost the same; a weighted or tree-aware tag distance might track human readability even more closely while keeping the pipeline interpretable.
  • The paper validates against preference, not comprehension; a reading-time or recall experiment on rendered pages would test the stated cognitive-load motivation directly.
  • The ground-truth-free, model-dependent reference design is portable to other style-oriented qualities, such as tone, citation formatting, or table use, where fixed ground truths are similarly unavailable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces MDEval, a benchmark for measuring the 'Markdown Awareness' of LLMs, defined as the structural quality of Markdown formatting in generated responses. The pipeline generates a response from a target LLM (Phase 1), rewrites that response using GPT-4o with a fixed prompt to create a model-specific reference (Phase 2), converts both outputs to HTML and extracts tags (Phase 3), and computes a normalized Levenshtein edit distance between tag sequences as the score (Phase 4, Eq. 1). The authors construct a 20K-instance dataset in English and Chinese across 10 subjects, evaluate 9 LLMs, and validate MDEval through a human pairwise-comparison study with Elo ratings. They report a record-level human-alignment accuracy of 84.1% and Spearman correlation 0.791 (Table 2), claiming to outperform existing methods by a large margin. They also show that QLoRA fine-tuning on a subset of the dataset improves Baichuan2-13B-chat-v1's MDEval score from 0.171 to about 0.73, which they describe as approximating GPT-4o-level performance.

Significance. If the validity claims hold, MDEval would be a useful, low-cost benchmark for a dimension of LLM output quality that is important for web chatbots and largely ignored by prior benchmarks. The paper provides a new dataset, an open-source implementation, and a human-alignment evaluation system, all of which are valuable community resources. The core idea of comparing HTML-tag edit distance rather than raw surface text is sensible and interpretable. However, the paper's central evidentiary claim—that MDEval aligns with human preferences substantially better than existing methods—is weakened by the selective reporting of ties-skipped accuracy, and the use of GPT-4o as both the rewrite judge and one of the evaluated models raises a circularity risk that is only partially addressed by the human study. The fine-tuning result is also overstated relative to the reported GPT-4o score. These issues are fixable but require additional analysis and experiments.

major comments (4)
  1. [§5.3, Table 2 vs. Appendix B.2, Table 4] The headline human-alignment result of 84.1% accuracy for MDEval is computed on the subset of pairwise comparisons where ties are skipped. Appendix B.2, Table 4, which includes ties, reports MDEval accuracy as 63.4%, below P-LLM's 65.4%. Figure 4 shows that ties constitute 20–38% of judgments, so the excluded data are substantial and non-random. The abstract and Section 5.3 claim that MDEval 'outperforms existing methods by a large margin,' but this conclusion is not supported on the full dataset. Please report the ties-included results as the primary analysis or provide a principled justification for excluding ties, and qualify the 'large margin' claim accordingly.
  2. [§3.1, Phase 2 and Eq. (1)] The reference used for scoring is a GPT-4o rewrite of the target model's own output, produced with a single fixed prompt at temperature 1. Since GPT-4o is also one of the evaluated models (Table 1), the metric may partly measure agreement with GPT-4o's particular rewriting style rather than general Markdown readability. The human-alignment study provides external grounding, but it does not eliminate this concern because the human study uses the same pairwise framework and the metric is still anchored to GPT-4o. Please report robustness checks with multiple judge LLMs (e.g., Claude or Gemini as the rewriter), and quantify the variance of scores due to the stochastic rewriting (temperature 1, Appendix B.5).
  3. [§5.5, Figure 7] The abstract and Section 5.5 claim that fine-tuning on the proposed dataset enables less performant open-source models to achieve 'comparable performance to GPT-4o.' However, Figure 7 shows the fine-tuned Baichuan2-13B-chat-v1 reaches an MDEval score of about 0.73, while Table 1 lists GPT-4o at 0.865 and GPT-4o-mini at 0.830. This is not comparable to GPT-4o. Additionally, the fine-tuned model is only evaluated with the MDEval metric, which itself relies on GPT-4o as the rewriter; no human evaluation of the fine-tuned model's outputs is provided. Please revise the claim and add human or at least multi-judge evaluation of the fine-tuned model.
  4. [§3.2, Eq. (6)] The record-level accuracy indicator in Eq. (6) treats a human tie as correct only if the two MDEval scores are exactly equal. Since MA scores are continuous (normalized edit distances), exact equality essentially never occurs, so every human tie is automatically counted as an error when ties are included. This explains the large drop from 84.1% to 63.4% and makes the ties-included accuracy a harsh and arguably inappropriate measure. A tolerant tie-handling rule (e.g., treating score differences below a threshold as ties) or a soft agreement metric should be defined and reported.
minor comments (7)
  1. [§3.1] Typo: 'textural' should be 'textual' in the sentence about content bias.
  2. [§5.5] Typo: 'fine-turning' should be 'fine-tuning'.
  3. [§5.1] The phrase 'count the number of Markdown enumerate' should likely be 'count the number of Markdown elements'.
  4. [Appendix B.4] The heading 'QR5' appears to be a typo for 'RQ5'.
  5. [Figure 4] The caption says 'at an early snapshot of A', but the text later states the tie ratio 'stays relatively constant as the amount of collected data grows.' Please clarify whether Figure 4 shows the final dataset or an early snapshot, and report the total number of pairwise judgments collected.
  6. [Appendix B.5] The claim that outputs at temperature 1 are 'stable generally' is surprising and should be supported by repeated-generation statistics, since temperature 1 typically implies high stochasticity.
  7. [§4.2] The dataset is described as ground-truth-free, but Section 4.2 states that fine-tuning uses 'the response generated by the most advanced LLM' as ground truth. Please clarify that the evaluation dataset is ground-truth-free while the fine-tuning set uses a model-generated reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central validity claim is grounded in external human judgments, not in the metric's own construction.

full rationale

MDEval's score (Eq. 1) is 1 minus normalized edit distance between a model's HTMLified output and an HTMLified GPT-4o rewrite of that same output, so the reference is model-dependent and GPT-4o's own score is partly an auto-similarity measure; the paper acknowledges this in Appendix B.5 ('responses produced by the judge model can be subject to further refinement or optimization by the model itself'). This is a design limitation, but it is not a circular derivation of the paper's central claim. The human-alignment numbers in Section 5.3 are tested against an independently collected set of human pairwise preferences A (Eqs. 5-6), and no MDEval parameter is fitted to those labels; the Elo parameters d=400 and K=10 are taken from Chatbot Arena [6], not tuned to maximize alignment. The Spearman and accuracy figures are therefore external evidence rather than consequences of the definition. The RQ4 fine-tuning result is likewise not circular: Section 4.2 states the SFT ground truth is 'the response generated by the most advanced LLM reported in our human alignment evaluation system' (i.e., human-preferred outputs), not the metric's GPT-4o reference. The ties-skipped reporting in Table 2 versus Table 4 is a selective-reporting and robustness concern, but it does not make an output variable equal to an input variable by construction. No load-bearing self-citation chain exists anywhere in the paper.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central MDEval score has no fitted parameters, but it depends on the choice of GPT-4o as judge, on the fixed rewrite prompt, and on the assumption that HTML tag edit distance captures readability. The human-alignment validation introduces hand-chosen Elo parameters and relies on human preference as ground truth.

free parameters (4)
  • Elo rating spread d = 400
    Hand-chosen in Section 3.2 based on tie ratios reported for Chatbot Arena; affects the Elo-based human alignment validation, not the MDEval score itself.
  • Elo update factor K = 10
    Hand-chosen in Section 3.2 together with d; affects the stability of Elo ratings used to validate the metric.
  • D-Rule decay factor gamma = 0.5
    Heuristic decay factor in the D-Rule baseline (Appendix A.1); not part of MDEval's central metric but used in one baseline comparison.
  • D-Rule element weights = 10 for headings/code/math/list/bold, 5 for others
    Assigned by heuristic rules in Appendix A.1 for the D-Rule baseline; not part of MDEval's metric.
assumptions (4)
  • domain assumption GPT-4o's rewrite with the fixed prompt is a valid reference for Markdown structure quality.
    Section 3.1 Phase 2. If the judge model's rewriting style does not reflect human readability preferences, all MDEval scores inherit that bias.
  • domain assumption Edit distance between extracted HTML tag sequences faithfully captures structural Markdown quality.
    Section 3.1 Phases 3 and 4. Content is discarded and only tag sequences are compared, so readability differences that depend on content or layout may be missed.
  • domain assumption Human pairwise preference is a valid gold standard for Markdown Awareness.
    Section 3.2. The human alignment test treats agreement with human votes as the criterion of metric validity.
  • domain assumption The Markdown-to-HTML conversion tools used handle standard Markdown and can be extended with a custom <math> tag for LaTeX elements.
    Section 3.1 Phase 3. The pipeline depends on the behavior of HTMLifying tools, which are not identified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models." pith.science (2026). https://pith.science/paper/7MBVEOA5

@misc{pith2026250115000,
  author       = {Pith},
  title        = {Pith review of: MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7MBVEOA5}},
  note         = {Machine review of arXiv:2501.15000}
}
read the original abstract

Large language models (LLMs) are expected to offer structured Markdown responses for the sake of readability in web chatbots (e.g., ChatGPT). Although there are a myriad of metrics to evaluate LLMs, they fail to evaluate the readability from the view of output content structure. To this end, we focus on an overlooked yet important metric -- Markdown Awareness, which directly impacts the readability and structure of the content generated by these language models. In this paper, we introduce MDEval, a comprehensive benchmark to assess Markdown Awareness for LLMs, by constructing a dataset with 20K instances covering 10 subjects in English and Chinese. Unlike traditional model-based evaluations, MDEval provides excellent interpretability by combining model-based generation tasks and statistical methods. Our results demonstrate that MDEval achieves a Spearman correlation of 0.791 and an accuracy of 84.1% with human, outperforming existing methods by a large margin. Extensive experimental results also show that through fine-tuning over our proposed dataset, less performant open-source models are able to achieve comparable performance to GPT-4o in terms of Markdown Awareness. To ensure reproducibility and transparency, MDEval is open sourced at https://github.com/SWUFE-DB-Group/MDEval-Benchmark.

Figures

Figures reproduced from arXiv: 2501.15000 by the authors.

Figure 2
Figure 2. GPT-4o-mini is able to output mathematical equa [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 1
Figure 1. Differences of Markdown Awareness of LLMs under the same input prompt. tasks, including coding [5], math [9], and multi-turn questions [19]. A large variety of evaluation metrics, covering answer relevancy, faithfulness, contextual recall, contextual precision, hallucination, toxicity, and bias [1], are introduced to fit different requirements in practical applications. However, to the best of our knowledge, there i… view at source ↗
Figure 3
Figure 3. The overall framework of MDEval. Given the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The count of ties vs. wins + losses at an early snap [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The Elo ratings and confidence intervals from hu [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Spearman correlation between Markdown Awareness and LLMs’ capabilities in the public leaderboard [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 8
Figure 8. Figure 8: Markdown Awareness in English and Chinese of each LLM. Generally, linguistic contexts have little influence on the score in MDEval. 6 Conclusion and Future Work In this paper, we focus on an overlooked yet important metric — Markdown Awareness, which directly affects t…
Figure 7
Figure 7. Figure 7: Model performance in terms of Markdown Awareness with the size of SFT instances. 5.6 RQ5: Across Subjects and Languages The performance and capabilities of LLMs frequently exhibit het￾erogeneity across diverse domains and linguistic contexts. In light of this variabili…
Figure 10
Figure 10. Figure 10: Pairwise win rate summary [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 9
Figure 9. Figure 9: Average win rate summary [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 23 canonical work pages

  1. [1]

    Confident AI. 2024. DeepEval. https://github.com/confident-ai/deepeval. Ac- cessed: 2024-09-04

    work page 2024

  2. [2]

    Hélder Antunes and Carla Teixeira Lopes. 2019. Readability of web content. In 2019 14th Iberian Conference on Information Systems and Technologies (CISTI) . IEEE, 1–4. MDEval: Evaluating and Enhancing Markdown Awareness in Large Language Models WWW ’25, April 28–May 2, 2025, Sydney, NSW, Australia

    work page 2019

  3. [3]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  4. [4]

    Rishi Bommasani, Percy Liang, and Tony Lee. 2023. Holistic evaluation of lan- guage models. Annals of the New York Academy of Sciences1525, 1 (2023), 140–146

    2023

  5. [5]

    Linzheng Chai, Shukai Liu, Jian Yang, Yuwei Yin, Ke Jin, Jiaheng Liu, Tao Sun, Ge Zhang, Changyu Ren, Hongcheng Guo, et al. 2024. McEval: Massively Multi- lingual Code Evaluation. arXiv preprint arXiv:2406.07436 (2024)

    arXiv 2024

  6. [6]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Forty-first International Conference on Machine Learning

    work page 2024

  7. [7]

    Lin Chin-Yew. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out, 2004

    2004

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 (2018)

    arXiv 2018

Show all 43 references
  1. [9]

    Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. 2024. Evaluating language models for mathematics through interac- tions. Proceedings of the National Academy of Sci...

    2024

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  3. [11]

    Ehsan Elahi, Ana María Iglesias Maqueda, and Jorge Luis Morato Lara. 2022. Web Readability Challenges. In Proceedings of the Computational Methods in Systems and Software. Springer, 446–454

    2022

  4. [12]

    Alexander R Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summa- rization evaluation. Transactions of the Association for Computational Linguistics 9 (2021), 391–409

    2021

  5. [13]

    Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as You Desire. In Proceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 6556–6576

    2024

  6. [14]

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383 (2024)

    2024 arXiv

  7. [15]

    James Hartley. 2013. Designing instructional text. Routledge

    2013

  8. [16]

    Nawal Iram, Shareena Zafar, and Rida Zahra. 2018. Web content readability evaluation using fuzzy logic. In 2018 International Conference on Advancements in Computational Sciences (ICACS). IEEE, 1–8

    2018

  9. [17]

    Tapas Kanungo and David Orr. 2009. Predicting the readability of short web summaries. In Proceedings of the Second ACM International Conference on Web Search and Data Mining . 202–211

    2009

  10. [18]

    Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. 2015. From word embeddings to document distances. In International conference on machine learn- ing. PMLR, 957–966

    2015

  11. [19]

    Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. 2024. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models. arXiv preprint arXiv:2401.16745 (2024)

    2024 arXiv

  12. [20]

    Yang Liu, Jiahuan Cao, Chongyu Liu, Kai Ding, and Lianwen Jin. 2024. Datasets for large language models: A comprehensive survey. arXiv preprint arXiv:2402.18041 (2024)

    2024 arXiv

  13. [21]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2511–2522

    2023

  14. [22]

    Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero- Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 9004–9017

    2023

  15. [23]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics 12 (2024), 933–949

    2024

  16. [24]

    John Morkes and Jakob Nielsen. 1997. Concise, scannable, and objective: How to write for the Web. Useit. com 51, 1 (1997), 1–17

    1997

  17. [25]

    Subhabrata (Subho) Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Chris Meek, Ahmed Awadallah, and Jianfeng Gao. 2021. CLUES: Few-Shot Learning Evaluation in Natural Language Understanding. In NeurIPS 2021

    2021

  18. [26]

    Kun-Peng Ning, Shuo Yang, Yu-Yang Liu, Jia-Yu Yao, Zhen-Hui Liu, Yu Wang, Ming Pang, and Li Yuan. 2024. Peer-review-in-LLMs: Automatic Evaluation Method for LLMs in Open-environment. arXiv preprint arXiv:2402.01830 (2024)

    2024 arXiv

  19. [27]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  20. [28]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . 7881–7892

    2020

  21. [29]

    Christopher Tensmeyer, Zoya Bylinski, Tianyuan Cai, Dave Miller, Ani Nenkova, Aleena Niklaus, and Shaun Wallace. 2023. Web Table Formatting Affects Readabil- ity on Mobile Devices. InProceedings of the ACM Web Conference 2023. 1334–1344

    2023

  22. [30]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  23. [31]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks ...

    2018

  24. [32]

    Wikipedia. [n. d.]. Elo rating system. https://en.wikipedia.org/wiki/Elo_rating_ system. Accessed: 2024-09-21

    2024

  25. [33]

    Tianyu Wu, Shizhu He, Jingping Liu, Siqi Sun, Kang Liu, Qing-Long Han, and Yang Tang. 2023. A brief overview of ChatGPT: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica 10, 5 (2023), 1122–1136

    2023

  26. [34]

    Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua

  27. [35]

    Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data 18, 6 (2024), 1–32

    2024

  28. [36]

    Mona Zamiri, Yao Qiang, Fedor Nikolaev, Dongxiao Zhu, and Alexander Kotov

  29. [37]

    Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and LOU Jian-Guang. 2023. Large Language Models Meet NL2Code: A Survey. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  30. [38]

    In Proceedings of the ACM on Web Conference 2024

    Benchmark and Neural Architecture for Conversational Entity Retrieval from a Knowledge Graph. In Proceedings of the ACM on Web Conference 2024 . 1519–1528

    2024

  31. [39]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2023), 46595–46623

    2023

  32. [40]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  33. [43]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: S...

    2024 arXiv

  34. [2020]

    In 8th International Conference on Learning Representations, ICLR 2020

    BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020

    2020

  35. [2024]

    In Proceedings of the ACM on Web Conference 2024

    Search-in-the-Chain: Interactively Enhancing Large Language Models with Search for Knowledge-intensive Tasks. In Proceedings of the ACM on Web Conference 2024. 1362–1373

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.