Pith. sign in

REVIEW 2 major objections 4 minor 86 references

Statistical Multicriteria Evaluation of LLM-Generated Text

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Human-written text survives a statistical multicriteria test against five decoding strategies.

desk verdict A legitimate adaptation of the GSD-front to decoding evaluation, weakened by author-supplied human ratings and a cardinal metric fitted to the same research line—worth refereeing, not desk-rejecting. read the letter →

arxiv 2506.18082 v2 pith:BOUSXS2S submitted 2025-06-22 cs.CL stat.AP

Esteban Garces Arias , Hannah Blocher , Julian Rodemann , Matthias A{ss}enmacher , Christoph Jansen This is my paper · ORCID
classification cs.CLstat.AP
keywords generalizedstochasticdominanceGSD-frontLLMtextevaluationdecodingstrategiesmulticriteriabenchmarkingordinalandcardinalqualitymetricsrobuststatisticalinferencehumanversusmachine
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper turns benchmarking of LLM decoding strategies into a statistical hypothesis test. It adapts the GSD-front framework, a partial-order way of comparing options across several quality dimensions, so that a strategy counts as competitive only if no other strategy dominates it on all dimensions at once. The data are 300 text continuations of 50 prompts from WikiText and WikiNews, generated by Qwen2.5-7B with five decoding strategies and compared against human-written completions using one cardinal score (Q*Text) and two ordinal human ratings. The paper's central finding is that at level $\alpha = 0.05$ the human completions survive: none of the five automatic strategies significantly dominates them, so H lies in the empirical GSD-front, and the verdict holds for up to one contaminated prompt. The larger claim is that multicriteria evaluation can avoid arbitrary metric weighting while still producing inferential, robustness-aware conclusions.

What carries the argument

The load-bearing object is the GSD-front, the set of strategies not strictly dominated under generalized stochastic dominance (GSD), a partial-order comparison that holds across all utility functions compatible with a mixed ordinal-cardinal quality structure. The structure is built from two relations on quality vectors: $R_1$ encodes 'at least as good on every metric,' and $R_2$ encodes preference-intensity comparisons on the cardinal metric given ordinal bounds. The test statistic is $D(S,S') = \inf_{u \in \mathcal{U}_{\mathcal{P}}} \left( \mathbb{E}_{\pi} u(\Phi(S,\cdot)) - \mathbb{E}_{\pi} u(\Phi(S',\cdot)) \right)$, the smallest expected-utility advantage of $S$ over $S'$ among all compatible utilities, and its empirical version is fed into a permutation test; the front test rejects $H_0$ if and only if every pairwise subtest rejects. This machinery is what lets the analysis mix Q*Text (cardinal) with two 5-point Likert ratings (ordinal) and quantify robustness to i.i.d. violations via contamination-sized p-value curves.

What would settle it

Replace the two author Likert ratings with ratings of the same 300 continuations by independent, non-author annotators who are blind to source, and rerun the five pairwise GSD tests; if for any decoding strategy the test then fails to reject dominance over human text at $\alpha = 0.05$, the paper's central claim about human text lying in the GSD-front is refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a decoding strategy can be admitted to or excluded from a GSD-front by statistical test, and that human-written completions pass that test against current automatic strategies. Concretely, the paper rejects $H_0: H \notin \mathrm{gsd}(\{H, BS, CS, TS, Tk, Tp\})$ at $\alpha = 0.05$ because all five pairwise GSD-comparison tests reject their nulls; it therefore concludes that human completion is not significantly outperformed by any automatic decoding strategy in the set and can be assumed to lie in the GSD-front. The paper presents this not as proof that humans write better text, but as evidence that human text is not redundant: there is no automatic strategy that is at least as good on every considered quality dimension for every utility representation compatible with the data. The same test, the paper argues, lets a researcher decide whether a newly proposed decoding strategy is competitive before investing further in it.

Load-bearing premise

The human quality ratings used as the two ordinal dimensions come from the paper's own authors, and the cardinal metric Q*Text was tuned in earlier work to match human ratings from the same research line; if those ratings already favor human-written text, the conclusion that human text is not dominated is largely built into the data.

Editorial extensions

If this is right

  • Any new decoding strategy can be certified as competitive only if it survives the GSD-front test against the existing set, replacing single-metric leaderboards with a statistical partial order.
  • Benchmark papers can report not just point estimates but a significance level and a contamination budget, the number of non-i.i.d. prompts that would overturn the verdict.
  • Human and automatic evaluations can be analyzed jointly without forcing ordinal scales into cardinal numbers, so no arbitrary weighting scheme is needed.
  • The framework is transferable to other generation tasks, such as summarization and translation, where multiple quality dimensions compete.
  • For the specific comparison studied, the result implies current decoding strategies have not yet reached a quality level that makes human-written completions redundant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural follow-up not pursued here: rerun the same GSD-front test with independent, non-author annotators; the paper's human-superiority result would be on firmer ground if the front membership of H survives that replacement.
  • An editorial inference: the requirement that all pairwise subtests reject makes front membership a conservative claim, so a strategy that fails the test is clearly dominated, while one that passes is merely not dominated, not necessarily best.
  • If benchmark suites adopt contamination bounds as standard reporting, comparisons across papers become more meaningful because readers learn how fragile each ranking is.
  • The framework could be turned around to compare decoding strategies against each other without a human reference, producing a partial-order answer to which decoder to use.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper adapts the Generalized Stochastic Dominance (GSD)-front framework, previously proposed for classifier selection, to the multicriteria evaluation of LLM decoding strategies. It evaluates human-written completions (H) against five decoding strategies (beam search, contrastive search, temperature sampling, top-k sampling, and top-p sampling) using three quality coordinates: one cardinal automatic metric (Q*Text) and two ordinal human Likert ratings. The statistical test for whether H lies in the GSD-front is taken from Jansen et al. (2024). The empirical results reject the null hypothesis that H is not in the GSD-front at alpha = 0.05, and a robustness analysis quantifies how the decision changes under contamination of up to a few prompts. The paper positions the framework as addressing single-metric evaluation, mismatched measurement scales, and lack of inferential guarantees in benchmarking.

Significance. If the methodology is accepted, the framework is a principled step beyond single-metric or ad-hoc weighted evaluation: it respects cardinal vs. ordinal scales, avoids arbitrary weighting, and provides simultaneous inferential statements with a robustness assessment. The authors also make code, data, and evaluation results publicly available, which strengthens reproducibility. The main significance is therefore methodological. The empirical demonstration, however, is weakened by the fact that the three quality dimensions are not independent: Q*Text was parameterized in prior work to maximize correlation with human ratings from the same research line (Appendix A.3), and the two ordinal ratings were provided by the authors themselves. This means the claim that human text lies on the GSD-front is not an independent confirmation of human-text quality, but partly a restatement of a single human-preference signal encoded in all three metrics. The paper's contribution is still valuable if reframed as a proof-of-concept, but the specific empirical conclusion is not robustly supported.

major comments (2)
  1. [Sections 3.1-3.2, Appendix A.3] The three evaluation dimensions are not independent sources of evidence. Q*Text's nine parameters theta = {w_i, mu_i, alpha_i} are optimized in Eq. (11) to maximize Spearman correlation with human ratings from Garces Arias et al. (2025a), i.e., the same research line and partially overlapping authorship as the present authors. The ordinal dimensions phi_2 and phi_3 are supplied by two of the paper's authors in a blind protocol. If all three coordinates encode the same human-preference construct, then the conclusion that H lies in the GSD-front is largely predetermined: a metric fitted to human preferences and two human raters are unlikely to jointly show that automatic strategies dominate human text. The Limitations section acknowledges expertise bias but does not mention the Q*Text fitting issue, which is more consequential. To support the claim of multicriteria evaluation, the authors should either (a) use human ratings from independent evaluators outside the research group, (b) demonstrate empirically that the three metrics are not redundant, e.g., by reporting rank correlations among phi_1, phi_2, phi_3 across the 300 texts, or (c) explicitly reframe the experiment as a methodological illustration rather than an empirical discovery about human text quality. As it stands, this circularity is load-bearing for the central empirical claim in Section 3.3.
  2. [Section 3.3, Figure 2 (right)] The robustness analysis shows that the joint test decision is preserved only when at most one contaminated prompt (out of 50) is allowed for four of the five pairwise comparisons, and at most three for the remaining comparison. This is a very small tolerance, especially given the paper's motivating concerns about dependence in human evaluation sessions and mixing two benchmark suites. While the authors honestly report this result, the abstract and introduction present the ability to 'quantify robustness' as a principal contribution; the quantification reveals that the empirical inference is fragile. The authors should prominently discuss this fragility as a limitation of the current evidence rather than a success of the framework, and ideally provide additional contamination scenarios (e.g., larger fractions, block contamination) to characterize how the decision degrades. This issue does not invalidate the statistical method, but it substantially tempers the strength of the empirical demonstration supporting claims (III+IV).
minor comments (4)
  1. [Section 3.2] There is a typo in the sentence describing R1: 'based on the ordinal evaluations (phi_1, phi_3)' should instead reference phi_2 and phi_3, since phi_1 is the cardinal Q*Text metric. The same sentence also mentions 'the cardinal evaluation from the automated metric Q*Text (phi_1)', creating an internal contradiction about which coordinates are ordinal.
  2. [Figure 2 caption] The caption refers to the 'rejection threshold (alpha = 0.05)' in the left panel; this is more precisely the critical value of the permutation test statistic, not a threshold on the data axis. Consider clarifying the terminology for readability.
  3. [Appendix A.4.2] The inter-rater agreement statistics (weighted Cohen's kappa = 0.324, Spearman = 0.518) are useful, but they are not connected to the multicriteria analysis. Reporting agreement separately and not discussing how rater disagreement affects the ordinal coordinates' reliability leaves an open question about the stability of the GSD-front conclusion under different rater weights.
  4. [Section 3.3] The statement 'we conclude that human completion is not significantly outperformed by any of the automatic decoding strategies' is a bit stronger than what the test establishes: the test provides evidence that H lies in the GSD-front at level 0.05, which means non-domination, not 'not significantly outperformed' in an absolute sense. The current phrasing is acceptable but could be more precise about the distinction between statistical evidence and the absence of evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the GSD-front conclusion is an empirical test outcome, not a construction; the metric calibration and author ratings are acknowledged inputs that do not by themselves force the result.

full rationale

The paper's central claim—H lies in gsd(S) at α=0.05—is an empirical test result, not a derived theorem. The GSD machinery (Eqs. 2–8) defines dominance from observed Φ vectors, and the test from Jansen et al. (2024) is external, peer-reviewed, and parameter-free with respect to this application. The potentially self-referential elements are φ1 = Q*Text, whose nine parameters were optimized in Eq. (11) to match prior human ratings, and φ2/φ3, which are ratings by two of the authors (Section 3.1). However, these are inputs to the evaluation: the paper does not define H's GSD-front membership in terms of Q*Text's fitted parameters or the authors' ratings. The pairwise tests could in principle have failed—an automatic strategy could dominate H on all three coordinates—and the paper explicitly acknowledges the evaluator-bias limitation in its Limitations section. No equation in the paper reduces the claimed result to its own inputs, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central claim rests on two fitted inputs: the Q*Text metric parameters (fitted to human ratings in prior work) and the hand-chosen decoding hyperparameters. It also relies on the validity of the cited GSD-front test and on several domain assumptions about metric scales and prompt sampling. No new entities are introduced.

free parameters (2)
  • Q*Text parameters (wi, mu_i, alpha_i for i=1,2,3) = Optimized in prior work via Spearman correlation to human ratings (Garces-Arias et al., 2025b); exact values not…
    The cardinal quality dimension phi_1 is Q*Text, whose nine parameters were fitted to human judgments. The central claim that H lies in the GSD-front depends on this fitted metric.
  • Decoding strategy hyperparameters = Beam width=5, contrastive search k=10 alpha=0.6, temperature=0.9, top-k k=50, top-p p=0.95
    These hand-chosen hyperparameters, taken from prior work (Garces Arias et al. 2025a), define the five competing strategies. The conclusion is specific to these configurations.
assumptions (5)
  • standard math The permutation test for the GSD-front hypothesis pair (H0, H1) is valid and consistent under the stated 'mild assumptions' from Jansen et al. 2024.
    Section 2.2 (after Eq. 7) invokes Jansen et al. 2024 for the test; the assumptions are not stated in this paper, so the reader must accept the cited theorem.
  • standard math The set U_P of utility representations compatible with the partial order P exactly captures all preference information consistent with the mixed-scale observations.
    Section 2.2, Eq. (4), defines utility representations; the framework assumes this set is a faithful summary of the ordinal and cardinal information.
  • domain assumption Q*Text is a valid cardinal quality measure for open-ended text generation.
    The paper uses Q*Text as the cardinal metric phi_1 without re-deriving its validity; it is treated as established in Appendix A.3.
  • domain assumption Human Likert ratings can be treated as ordinal metrics phi_2 and phi_3, with meaningful comparisons only in sign, not magnitude.
    Section 3.2 assigns the two author ratings to the ordinal part of the metric space; this is a modeling assumption about human evaluation data.
  • domain assumption Prompts are generated from an underlying population pi, and any deviation from i.i.d. can be modeled by a small number of contaminated prompts.
    Section 3.3 interprets the robustness plot as p-values under 'contamination size'; this contamination model is an assumption about how real-world dependencies arise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Statistical Multicriteria Evaluation of LLM-Generated Text." pith.science (2026). https://pith.science/paper/BOUSXS2S

@misc{pith2026250618082,
  author       = {Pith},
  title        = {Pith review of: Statistical Multicriteria Evaluation of LLM-Generated Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BOUSXS2S}},
  note         = {Machine review of arXiv:2506.18082}
}
read the original abstract

Assessing the quality of LLM-generated text remains a fundamental challenge in natural language processing. Current evaluation approaches often rely on isolated metrics or simplistic aggregations that fail to capture the nuanced trade-offs between coherence, diversity, fluency, and other relevant indicators of text quality. In this work, we adapt a recently proposed framework for statistical inference based on Generalized Stochastic Dominance (GSD) that addresses three critical limitations in existing benchmarking methodologies: the inadequacy of single-metric evaluation, the incompatibility between cardinal automatic metrics and ordinal human judgments, and the lack of inferential statistical guarantees. The GSD-front approach enables simultaneous evaluation across multiple quality dimensions while respecting their different measurement scales, building upon partial orders of decoding strategies, thus avoiding arbitrary weighting of the involved metrics. By applying this framework to evaluate common decoding strategies against human-generated text, we demonstrate its ability to identify statistically significant performance differences while accounting for potential deviations from the i.i.d. assumption of the sampling design.

Figures

Figures reproduced from arXiv: 2506.18082 by the authors.

Figure 1
Figure 1. Analysis of text generation research trends (arXiv, 2022-2024): (a) Evolution of multicrite [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Left: Empirical densities of resampled test statistics for pairwise GSD-comparisons between [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Evaluation results for 300 text continuations generated from 50 prompts derived from [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

86 extracted references · 47 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    A learning algorithm for boltzmann machines

    David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 9 0 (1): 0 147--169, 1985

    1985

  3. [3]

    Bayesian optimization for building social-influence-free consensus

    Masaki Adachi, Siu Lun Chau, Wenjie Xu, Anurag Singh, Michael A Osborne, and Krikamol Muandet. Bayesian optimization for building social-influence-free consensus. arXiv preprint arXiv:2502.07166, 2025

    arXiv 2025

  4. [4]

    Jointly measuring diversity and quality in text generation models

    Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah. Jointly measuring diversity and quality in text generation models. In Antoine Bosselut, Asli Celikyilmaz, Marjan Ghazvininejad, Srinivasan Iyer, Urvashi Khandelwal, Hannah Rashkin, and Thomas Wolf (eds.), Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Languag...

    doi:10.18653/v1/w19-2311 2019

  5. [5]

    Text generation: A systematic literature review of tasks, evaluation, and challenges, 2024

    Jonas Becker, Jan Philip Wahle, Bela Gipp, and Terry Ruas. Text generation: A systematic literature review of tasks, evaluation, and challenges, 2024. URL https://arxiv.org/abs/2405.15604

    arXiv 2024

  6. [6]

    Howcroft

    Anya Belz, Simon Mille, and David M. Howcroft. Disentangling the properties of human evaluation methods: A classification system to support comparability, meta-evaluation and reproducibility testing. In Brian Davis, Yvette Graham, John Kelleher, and Yaji Sripada (eds.), Proceedings of the 13th International Conference on Natural Language Generation, pp.\ ...

    doi:10.18653/v1/2020.inlg-1.24 2020

  7. [7]

    Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis

    Alessio Benavoli, Giorgio Corani, Janez Dem s ar, and Marco Zaffalon. Time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. Journal of Machine Learning Research, 18 0 (77): 0 1--36, 2017

    2017

  8. [8]

    Estimating the replication probability of significant classification benchmark experiments

    Daniel Berrar. Estimating the replication probability of significant classification benchmark experiments. Journal of Machine Learning Research, 25 0 (311): 0 1--42, 2024

    2024

Show all 86 references
  1. [9]

    Comparing machine learning algorithms by union-free generic depth

    Hannah Blocher, Georg Schollmeyer, Malte Nalenz, and Christoph Jansen. Comparing machine learning algorithms by union-free generic depth. International Journal of Approximate Reasoning, 169: 0 109166, 2024

    2024

  2. [10]

    How to choose a reinforcement-learning algorithm

    Fabian Bongratz, Vladimir Golkov, Lukas Mautner, Luca Della Libera, Frederik Heetmeyer, Felix Czaja, Julian Rodemann, and Daniel Cremers. How to choose a reinforcement-learning algorithm. arXiv preprint arXiv:2407.20917, 2024

    2024 arXiv

  3. [11]

    Self-learning from pairwise credal labels

    Vitor Martin Bordini, S \'e bastien Destercke, and Benjamin Quost. Self-learning from pairwise credal labels. In 27th European Conference on Artificial Intelligence (ECAI 2024), 2024

    2024

  4. [12]

    Credal bayesian deep learning

    Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal bayesian deep learning. arXiv preprint arXiv:2302.09656, 2023

    2023 arXiv

  5. [13]

    Evaluation of text generation: A survey, 2021

    Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey, 2021. URL https://arxiv.org/abs/2006.14799

    2021 arXiv

  6. [14]

    Evaluating language models as risk scores

    Andr \'e F Cruz, Moritz Hardt, and Celestine Mendler-D \"u nner. Evaluating language models as risk scores. arXiv preprint arXiv:2407.14614, 2024

    2024 arXiv

  7. [15]

    Dem s ar

    J. Dem s ar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7: 0 1--30, 2006

    2006

  8. [16]

    Semi-supervised learning guided by the generalized bayes rule under soft revision

    Stefan Dietrich, Julian Rodemann, and Christoph Jansen. Semi-supervised learning guided by the generalized bayes rule under soft revision. In International Conference on Soft Methods in Probability and Statistics, pp.\ 110--117. Springer, 2024

    2024

  9. [17]

    R. Dror, A. Ordan, et al. A hitchhiker’s guide to statistical significance testing in nlp. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 27--33, 2018

    2018

  10. [18]

    Eugster, T

    M. Eugster, T. Hothorn, and F. Leisch. Domain-based benchmark experiments: Exploratory and inferential analysis. Austrian Journal of Statistics, 41 0 (1): 0 5--26, 2012

    2012

  11. [19]

    Hierarchical neural story generation, 2018

    Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation, 2018

    2018

  12. [20]

    Beam search strategies for neural machine translation

    Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. Association for Computational Linguistics, 2017. doi:10.18653/v1/w17-3207. URL http://dx.doi.org/10.18653/v1/W17-3207

    2017 doi

  13. [21]

    Simcse: Simple contrastive learning of sentence embeddings, 2022

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings, 2022

    2022

  14. [22]

    Adaptive contrastive search: Uncertainty-guided decoding for open-ended text generation, 2024

    Esteban Garces-Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, and Matthias Aßenmacher. Adaptive contrastive search: Uncertainty-guided decoding for open-ended text generation, 2024. URL https://arxiv.org/abs/2407.18698

    2024 arXiv

  15. [23]

    Decoding decoded: Understanding hyperparameter effects in open-ended text generation

    Esteban Garces Arias, Meimingwei Li, Christian Heumann, and Matthias Assenmacher. Decoding decoded: Understanding hyperparameter effects in open-ended text generation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (...

    2025

  16. [24]

    Towards better open-ended text generation: A multicriteria evaluation framework, 2025b

    Esteban Garces-Arias, Hannah Blocher, Julian Rodemann, Meimingwei Li, Christian Heumann, and Matthias Aßenmacher. Towards better open-ended text generation: A multicriteria evaluation framework, 2025b. URL https://arxiv.org/abs/2410.18653

    arXiv

  17. [25]

    Garc \'i a and F

    S. Garc \'i a and F. Herrera. An extension on `` S tatistical comparisons of classifiers over multiple cata sets'' for all pairwise comparisons. Journal of Machine Learning Research, 9: 0 2677--2694, 2008

    2008

  18. [26]

    García, A

    S. García, A. Fernández, J. Luengo, and F. Herrera. Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences, 180 0 (10): 0 2044--2064, 2010

    2010

  19. [27]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  20. [28]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  21. [29]

    Hashimoto, Hugh Zhang, and Percy Liang

    Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation, 2019. URL https://arxiv.org/abs/1904.02792

    2019 arXiv

  22. [30]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019

    1904 arXiv

  23. [31]

    Hothorn, F

    T. Hothorn, F. Leisch, A. Zeileis, and K. Hornik. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics, 14 0 (3): 0 675--699, 2005

    2005

  24. [32]

    Open graph benchmark: Datasets for machine learning on graphs

    Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33: 0 22118--22133, 2020

    2020

  25. [33]

    P. Huber. Robust Statistics. Wiley, New York, 1981

    1981

  26. [34]

    Jansen, G

    C. Jansen, G. Schollmeyer, and T. Augustin. Concepts for decision making under severe uncertainty with partial ordinal and partial cardinal preferences. International Journal of Approximate Reasoning, 98: 0 112--131, 2018

    2018

  27. [35]

    Jansen, H

    C. Jansen, H. Blocher, T. Augustin, and G. Schollmeyer. Information efficient learning of complexly structured preferences: Elicitation procedures and their application to decision making under uncertainty. International Journal of Approximate Reasoning, 144: 0 69--91, 2022

    2022

  28. [36]

    Contributions to the decision theoretic foundations of machine learning and robust statistics under weakly structured information, 2025

    Christoph Jansen. Contributions to the decision theoretic foundations of machine learning and robust statistics under weakly structured information, 2025. URL https://arxiv.org/abs/2501.10195

    2025 arXiv

  29. [37]

    Statistical comparisons of classifiers by generalized stochastic dominance

    Christoph Jansen, Malte Nalenz, Georg Schollmeyer, and Thomas Augustin. Statistical comparisons of classifiers by generalized stochastic dominance. Journal of Machine Learning Research, 24 0 (231): 0 1--37, 2023 a

    2023

  30. [38]

    Multi-target decision making under conditions of severe uncertainty

    Christoph Jansen, Georg Schollmeyer, and Thomas Augustin. Multi-target decision making under conditions of severe uncertainty. In Vicen c Torra and Yasuo Narukawa (eds.), Modeling Decisions for Artificial Intelligence, pp.\ 45--57, Cham, 2023 b . Springer Nature Switzerland

    2023

  31. [39]

    Robust statistical comparison of random variables with locally varying scale of measurement

    Christoph Jansen, Georg Schollmeyer, Hannah Blocher, Julian Rodemann, and Thomas Augustin. Robust statistical comparison of random variables with locally varying scale of measurement. In Uncertainty in Artificial Intelligence, pp.\ 941--952. PMLR, 2023 c

    2023

  32. [40]

    Statistical multicriteria benchmarking via the GSD -front

    Christoph Jansen, Georg Schollmeyer, Julian Rodemann, Hannah Blocher, and Thomas Augustin. Statistical multicriteria benchmarking via the GSD -front. Advances in Neural Information Processing Systems, 2024

    2024

  33. [41]

    Jelinek, R

    F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. Perplexity—a measure of the difficulty of speech recognition tasks . The Journal of the Acoustical Society of America, 62 0 (S1): 0 S63--S63, 08 2005. ISSN 0001-4966. doi:10.1121/1.2016299. URL https://doi.org/10.1121/1.2016299

    2005 doi

  34. [42]

    The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models

    Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about...

    2024

  35. [43]

    Efficient multi-criteria optimization on noisy machine learning problems

    Patrick Koch, Tobias Wagner, Michael TM Emmerich, Thomas B \"a ck, and Wolfgang Konen. Efficient multi-criteria optimization on noisy machine learning problems. Applied Soft Computing, 29: 0 357--370, 2015

    2015

  36. [44]

    Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning

    Ravin Kohli, Matthias Feurer, Katharina Eggensperger, Bernd Bischl, and Frank Hutter. Towards quantifying the effect of datasets for benchmarking: A look at tabular machine learning. 2024

    2024

  37. [45]

    Accelerated experimental design using a human-ai teaming framework

    Arun AV Kumar, Alistair Shilton, Sunil Gupta, Shannon Ryan, Majid Abdolshah, Hung Le, Santu Rana, Julian Berk, Mahad Rashid, and Svetha Venkatesh. Accelerated experimental design using a human-ai teaming framework. Knowledge-Based Systems, pp.\ 113138, 2025

    2025

  38. [46]

    Factuality enhanced language models for open-ended text generation

    Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35: 0 34586--34599, 2022

    2022

  39. [47]

    A diversity-promoting objective function for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language ...

    2016

  40. [48]

    Contrastive decoding: Open-ended text generation as optimization, 2023

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization, 2023

    2023

  41. [49]

    Quantifying variance in evaluation benchmarks

    Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint arXiv:2406.10229, 2024

    2024 arXiv

  42. [50]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  43. [51]

    Mersmann, M

    O. Mersmann, M. Preuss, H. Trautmann, B. Bischl, and C. Weihs. Analyzing the BBOB results by means of benchmarking concepts. Evolutionary Computation, 23: 0 161--185, 2015

    2015

  44. [52]

    Meyer, F

    D. Meyer, F. Leisch, and K. Hornik. The support vector machine under test. Neurocomputing, 55 0 (1): 0 169--186, 2003

    2003

  45. [53]

    Learning de-biased regression trees and forests from complex samples

    Malte Nalenz, Julian Rodemann, and Thomas Augustin. Learning de-biased regression trees and forests from complex samples. Machine Learning, 113 0 (6): 0 3379--3398, 2024

    2024

  46. [54]

    S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, and M. Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13 0 (1): 0 6793, 2022

    2022

  47. [55]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34: 0 4816--4828, 2021

    2021

  48. [56]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  49. [57]

    Partial rankings of optimizers

    Julian Rodemann and Hannah Blocher. Partial rankings of optimizers. In International Conference on Learning Representations (ICLR), Tiny Papers Track, 2024

    2024

  50. [58]

    Levelwise data disambiguation by cautious superset classification

    Julian Rodemann, Dominik Kreiss, Eyke H \"u llermeier, and Thomas Augustin. Levelwise data disambiguation by cautious superset classification. In International Conference on Scalable Uncertainty Management, pp.\ 263--276. Springer, 2022

    2022

  51. [59]

    Approximately bayes-optimal pseudo-label selection

    Julian Rodemann, Jann Goschenhofer, Emilio Dorigatti, Thomas Nagler, and Thomas Augustin. Approximately bayes-optimal pseudo-label selection. In Uncertainty in Artificial Intelligence, pp.\ 1762--1773. PMLR, 2023 a

    2023

  52. [60]

    In all likelihoods: Robust selection of pseudo-labeled data

    Julian Rodemann, Christoph Jansen, Georg Schollmeyer, and Thomas Augustin. In all likelihoods: Robust selection of pseudo-labeled data. In International Symposium on Imprecise Probability: Theories and Applications, pp.\ 412--425. PMLR, 2023 b

    2023

  53. [61]

    Explaining bayesian optimization by shapley values facilitates human-ai collaboration

    Julian Rodemann, Federico Croppi, Philipp Arens, Yusuf Sale, Julia Herbinger, Bernd Bischl, Eyke H \"u llermeier, Thomas Augustin, Conor J Walsh, and Giuseppe Casalicchio. Explaining bayesian optimization by shapley values facilitates human-ai collaboration. arXiv preprint arX...

    2024 arXiv

  54. [62]

    A statistical case against empirical human-ai alignment, 2025

    Julian Rodemann, Esteban Garces Arias, Christoph Luther, Christoph Jansen, and Thomas Augustin. A statistical case against empirical human-ai alignment, 2025. URL https://arxiv.org/abs/2502.14581

    2025 arXiv

  55. [63]

    A meta-analysis of overfitting in machine learning

    Rebecca Roelofs, Vaishaal Shankar, Benjamin Recht, Sara Fridovich-Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32, 2019

    2019

  56. [64]

    Schneider, L

    F. Schneider, L. Balles, and P. Hennig. DeepOBS : A deep learning optimizer benchmark suite. In International Conference on Learning Representations, 2018

    2018

  57. [65]

    A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes

    Chengchun Shi, Masatoshi Uehara, Jiawei Huang, and Nan Jiang. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning, pp.\ 20057--20094. PMLR, 2022

    2022

  58. [66]

    Shirali, R

    A. Shirali, R. Abebe, and M. Hardt. A theory of dynamic benchmarks. In The Eleventh International Conference on Learning Representations, 2023

    2023

  59. [67]

    An empirical study on contrastive search and contrastive decoding for open-ended text generation, 2022

    Yixuan Su and Jialu Xu. An empirical study on contrastive search and contrastive decoding for open-ended text generation, 2022

    2022

  60. [68]

    A contrastive framework for neural text generation, 2022

    Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation, 2022

    2022

  61. [69]

    Evaluating the evaluation of diversity in natural language generation

    Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Vol...

    2021

  62. [70]

    Scientific machine learning benchmarks

    Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, and Tony Hey. Scientific machine learning benchmarks. Nature Reviews Physics, 4 0 (6): 0 413--420, 2022

    2022

  63. [71]

    Openml: networked science in machine learning

    Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo. Openml: networked science in machine learning. ACM SIGKDD Explorations Newsletter, 15 0 (2): 0 49--60, 2014

    2014

  64. [72]

    P. Walley. Statistical Reasoning with Imprecise Probabilities. Chapman and Hall, London, 1991

    1991

  65. [73]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  66. [74]

    Principled bayesian optimisation in collaboration with human experts

    Wenjie Xu, Masaki Adachi, Colin N Jones, and Michael A Osborne. Principled bayesian optimisation in collaboration with human experts. arXiv preprint arXiv:2410.10452, 2024

    2024 arXiv

  67. [75]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  68. [76]

    Benchmarking llms via uncertainty quantification

    Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaopeng Tu. Benchmarking llms via uncertainty quantification. Advances in Neural Information Processing Systems, 37: 0 15356--15385, 2024

    2024

  69. [77]

    Zhang and M

    G. Zhang and M. Hardt. Inherent trade-offs between diversity and stability in multi-task benchmark, 2024 a

    2024

  70. [78]

    Inherent trade-offs between diversity and stability in multi-task benchmarks

    Guanhua Zhang and Moritz Hardt. Inherent trade-offs between diversity and stability in multi-task benchmarks. In International Conference on Machine Learning, 2024 b

    2024

  71. [79]

    Zhang, M

    J. Zhang, M. Harman, L. Ma, and Y. Liu. Machine learning testing: Survey, landscapes and horizons. IEEE Transactions on Software Engineering, 48 0 (1): 0 1--36, 2020

    2020

  72. [80]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022 arXiv

  73. [81]

    Don't make your llm an evaluation benchmark cheater

    Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023

    2023 arXiv

  74. [82]

    Time-varying gaussian process bandits with unknown prior

    Juliusz Ziomek, Masaki Adachi, and Michael A Osborne. Time-varying gaussian process bandits with unknown prior. arXiv preprint arXiv:2402.01632, 2024

    2024 arXiv

  75. [83]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  76. [84]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  77. [85]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  78. [86]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.