{"id":"ca81bbf8-cd93-4736-a79f-d994bd8c4493","arxiv_id":"2506.18082","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using generalized stochastic dominance, the authors find that human-written text completions are not significantly outperformed by five LLM decoding strategies across mixed cardinal and ordinal quality metrics.","lead":"This paper applies a statistical ranking framework called the GSD-front to compare LLM decoding strategies against human-written text on several quality metrics at once. It reports that human completions are not dominated by any of five common decoding strategies, and it checks how robust that conclusion is when the i.i.d. assumption is slightly violated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical claim that human text lies in the GSD-front rests on evaluation dimensions that are not independent: the ordinal ratings are author-supplied and the cardinal metric Q*Text was fitted to human ratings from the same research line.","rationale":"The GSD-front adaptation is a legitimate methodological contribution: it preserves mixed measurement scales, avoids arbitrary metric weights, and its inferential machinery is inherited from a published NeurIPS paper. The robustness analysis in Figure 2 is an honest quantification of the fragility of the decision. The problem identified here is not the mathematics but the evidence base for the empirical headline claim. All three quality dimensions are connected to the same human-preference source: Q*Text was calibrated against human ratings from prior work by the same research line, and the ordinal dimensions are author ratings. The test therefore cannot cleanly separate 'human text is objectively on the frontier' from 'the evaluation dimensions were designed, or produced, in a way that places human text on the frontier.' An independent-rater re-run, together with a non-fitted automatic metric, would settle whether this concern actually lands. This matches the reader's weakest_assumption, so the CONDITIONAL verdict should stand without adjustment.","tokens_in":19263,"tokens_out":9032,"duration_ms":104656,"concrete_test":"Run the identical Section 3.3 analysis on the same 50 prompts / 300 continuations with (a) φ2 and φ3 replaced by ratings from at least three independent non-author annotators using the Appendix A.4.1 instructions, and (b) φ1 replaced by the unoptimized equal-weight version of Eq. 10 (all wi = 1, no Gaussian penalties). If, in either variant, at least one of the five pairwise GSD comparisons fails to reject at α = 0.05, the headline front membership depends on the potentially circular evaluation inputs. If all five pairwise tests still reject under both variants, the concern is substantially resolved. Report the resulting p-values and inter-rater agreement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3's headline inference—H ∈ gsd(S) at α = 0.05—is established through three quality coordinates: φ1 = Q*Text (cardinal), φ2 and φ3 = Likert ratings by two of the paper's authors (ordinal). The conclusion is only as strong as the independence of these coordinates. Q*Text's parameters are optimized in Appendix A.3 (Eq. 11) to maximize Spearman correlation with human ratings from Garces-Arias et al. (2025a), which is the same research line and partly overlapping authorship. The two ordinal coordinates are supplied by the authors themselves (Section 3.1). The blind protocol mitigates but does not remove the risk that all three dimensions encode a common preference favoring human-written continuations. Under that scenario, 'H is not significantly dominated by any automatic strategy' is close to predetermined, because the evaluation dimensions themselves were constructed or produced by people with a stake in human-like text quality. The paper itself acknowledges this expertise-bias limitation in its Limitations section, so this is an external-validity and potential-circularity risk rather than an internal contradiction. The GSD machinery may be sound, but the empirical demonstration does not independently support the claim that human completions lie on the GSD-front.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper adapts the Generalized Stochastic Dominance (GSD)-front framework, previously proposed for classifier selection, to the multicriteria evaluation of LLM decoding strategies. It evaluates human-written completions (H) against five decoding strategies (beam search, contrastive search, temperature sampling, top-k sampling, and top-p sampling) using three quality coordinates: one cardinal automatic metric (Q*Text) and two ordinal human Likert ratings. The statistical test for whether H lies in the GSD-front is taken from Jansen et al. (2024). The empirical results reject the null hypothesis that H is not in the GSD-front at alpha = 0.05, and a robustness analysis quantifies how the decision changes under contamination of up to a few prompts. The paper positions the framework as addressing single-metric evaluation, mismatched measurement scales, and lack of inferential guarantees in benchmarking.","tokens_in":19488,"tokens_out":7940,"duration_ms":75683,"significance":"If the methodology is accepted, the framework is a principled step beyond single-metric or ad-hoc weighted evaluation: it respects cardinal vs. ordinal scales, avoids arbitrary weighting, and provides simultaneous inferential statements with a robustness assessment. The authors also make code, data, and evaluation results publicly available, which strengthens reproducibility. The main significance is therefore methodological. The empirical demonstration, however, is weakened by the fact that the three quality dimensions are not independent: Q*Text was parameterized in prior work to maximize correlation with human ratings from the same research line (Appendix A.3), and the two ordinal ratings were provided by the authors themselves. This means the claim that human text lies on the GSD-front is not an independent confirmation of human-text quality, but partly a restatement of a single human-preference signal encoded in all three metrics. The paper's contribution is still valuable if reframed as a proof-of-concept, but the specific empirical conclusion is not robustly supported.","major_comments":[{"comment":"The three evaluation dimensions are not independent sources of evidence. Q*Text's nine parameters theta = {w_i, mu_i, alpha_i} are optimized in Eq. (11) to maximize Spearman correlation with human ratings from Garces Arias et al. (2025a), i.e., the same research line and partially overlapping authorship as the present authors. The ordinal dimensions phi_2 and phi_3 are supplied by two of the paper's authors in a blind protocol. If all three coordinates encode the same human-preference construct, then the conclusion that H lies in the GSD-front is largely predetermined: a metric fitted to human preferences and two human raters are unlikely to jointly show that automatic strategies dominate human text. The Limitations section acknowledges expertise bias but does not mention the Q*Text fitting issue, which is more consequential. To support the claim of multicriteria evaluation, the authors should either (a) use human ratings from independent evaluators outside the research group, (b) demonstrate empirically that the three metrics are not redundant, e.g., by reporting rank correlations among phi_1, phi_2, phi_3 across the 300 texts, or (c) explicitly reframe the experiment as a methodological illustration rather than an empirical discovery about human text quality. As it stands, this circularity is load-bearing for the central empirical claim in Section 3.3.","section":"Sections 3.1-3.2, Appendix A.3"},{"comment":"The robustness analysis shows that the joint test decision is preserved only when at most one contaminated prompt (out of 50) is allowed for four of the five pairwise comparisons, and at most three for the remaining comparison. This is a very small tolerance, especially given the paper's motivating concerns about dependence in human evaluation sessions and mixing two benchmark suites. While the authors honestly report this result, the abstract and introduction present the ability to 'quantify robustness' as a principal contribution; the quantification reveals that the empirical inference is fragile. The authors should prominently discuss this fragility as a limitation of the current evidence rather than a success of the framework, and ideally provide additional contamination scenarios (e.g., larger fractions, block contamination) to characterize how the decision degrades. This issue does not invalidate the statistical method, but it substantially tempers the strength of the empirical demonstration supporting claims (III+IV).","section":"Section 3.3, Figure 2 (right)"}],"minor_comments":[{"comment":"There is a typo in the sentence describing R1: 'based on the ordinal evaluations (phi_1, phi_3)' should instead reference phi_2 and phi_3, since phi_1 is the cardinal Q*Text metric. The same sentence also mentions 'the cardinal evaluation from the automated metric Q*Text (phi_1)', creating an internal contradiction about which coordinates are ordinal.","section":"Section 3.2"},{"comment":"The caption refers to the 'rejection threshold (alpha = 0.05)' in the left panel; this is more precisely the critical value of the permutation test statistic, not a threshold on the data axis. Consider clarifying the terminology for readability.","section":"Figure 2 caption"},{"comment":"The inter-rater agreement statistics (weighted Cohen's kappa = 0.324, Spearman = 0.518) are useful, but they are not connected to the multicriteria analysis. Reporting agreement separately and not discussing how rater disagreement affects the ordinal coordinates' reliability leaves an open question about the stability of the GSD-front conclusion under different rater weights.","section":"Appendix A.4.2"},{"comment":"The statement 'we conclude that human completion is not significantly outperformed by any of the automatic decoding strategies' is a bit stronger than what the test establishes: the test provides evidence that H lies in the GSD-front at level 0.05, which means non-domination, not 'not significantly outperformed' in an absolute sense. The current phrasing is acceptable but could be more precise about the distinction between statistical evidence and the absence of evidence.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is methodologically interesting but the empirical illustration is compromised by the non-independence of the evaluation dimensions: Q*Text was fitted to human ratings from the same research group and the only human ratings come from the authors. This is not a fatal flaw for the framework, but it is a serious weakness in the demonstration that needs to be addressed or substantially reframed. The robustness result is also notably fragile (at most one contaminated prompt), which should be discussed candidly. Given the availability of code/data and the soundness of the underlying GSD theory, I see a clear path to revision, but the current version is not acceptable as-is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does two things: it ports the GSD-front framework from classifier selection to LLM decoding evaluation, and it runs a small study claiming human completions lie on the empirical GSD-front against five decoding strategies at alpha=0.05, robust to one contaminated prompt. The first is genuinely useful. It gives the field a way to evaluate decoding strategies on mixed cardinal/ordinal metrics without arbitrary weighting, and the statistical test is inherited from Jansen et al. 2024, which is fine—they cite it and apply it carefully. The second is where I have concerns.\n\nWhat's new: the specific result about human text is not in the literature, and the robustness plot (p-values vs contamination size) is a nice practical touch. They also ship code and data, which is real credit. The framework extension itself is coherent, and the paper is honest about several limitations.\n\nThe soft spot is load-bearing, not minor. The ordinal ratings come from two of the authors (Section 3.1), and the cardinal metric Q*Text was optimized in Eq. (11) to maximize Spearman correlation with human ratings from the same group's prior work (Garces-Arias et al. 2025a). So the three quality coordinates are not independent; all three could encode a common preference for human-like text. If that is the case, the conclusion that H is not dominated is largely predetermined. The paper acknowledges the author-rating limitation in its Limitations section, but it does not fully confront the deeper circularity: the metric used as the sole cardinal coordinate was fitted to ratings produced by the same research line, and the human raters are part of that line. The GSD machinery itself is sound—this is an external-validity problem, not an internal contradiction—but it means the empirical demonstration does not independently support the headline claim.\n\nThere is also a minor sample issue: 50 prompts, one model (Qwen 2.5-7B), and inter-rater agreement is only fair (weighted kappa 0.324). The robustness result (at most one contaminated prompt) is honest but weak, which the paper states clearly.\n\nWho this is for: people building decoding benchmarks or working on multi-metric evaluation. They will find the GSD-front adaptation useful regardless of the specific human-text result. It deserves a serious referee. I would send it out with a clear request for major revision: independent human ratings (even a small crowd), a transparent statement of Q*Text parameter provenance, and a sensitivity analysis that drops Q*Text or uses a metric not fitted to human preference. With those changes, the paper would be a solid contribution.\n\nRecommendation: engage with it. This is a desk-review decision, not a reject.","headline":"A legitimate adaptation of the GSD-front to decoding evaluation, weakened by author-supplied human ratings and a cardinal metric fitted to the same research line—worth refereeing, not desk-rejecting.","tokens_in":20066,"tokens_out":1804,"would_cite":true,"duration_ms":18828,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human-written text survives a statistical multicriteria test against five decoding strategies.","keywords":["generalized stochastic dominance","GSD-front","LLM text evaluation","decoding strategies","multicriteria benchmarking","ordinal and cardinal quality metrics","robust statistical inference","human versus machine text"],"falsifier":"Replace the two author Likert ratings with ratings of the same 300 continuations by independent, non-author annotators who are blind to source, and rerun the five pairwise GSD tests; if for any decoding strategy the test then fails to reject dominance over human text at $\\alpha = 0.05$, the paper's central claim about human text lying in the GSD-front is refuted.","tokens_in":19039,"feed_emoji":"📊","tokens_out":8497,"duration_ms":76093,"temperature":0.7,"pith_summary":"This paper turns benchmarking of LLM decoding strategies into a statistical hypothesis test. It adapts the GSD-front framework, a partial-order way of comparing options across several quality dimensions, so that a strategy counts as competitive only if no other strategy dominates it on all dimensions at once. The data are 300 text continuations of 50 prompts from WikiText and WikiNews, generated by Qwen2.5-7B with five decoding strategies and compared against human-written completions using one cardinal score (Q*Text) and two ordinal human ratings. The paper's central finding is that at level $\\alpha = 0.05$ the human completions survive: none of the five automatic strategies significantly dominates them, so H lies in the empirical GSD-front, and the verdict holds for up to one contaminated prompt. The larger claim is that multicriteria evaluation can avoid arbitrary metric weighting while still producing inferential, robustness-aware conclusions.","feed_headline":"Human text holds its own against five LLM decoding strategies","feed_subtitle":"A permutation test on 50 prompts finds no automatic decoder significantly dominates human completions at α = 0.05.","key_machinery":"The load-bearing object is the GSD-front, the set of strategies not strictly dominated under generalized stochastic dominance (GSD), a partial-order comparison that holds across all utility functions compatible with a mixed ordinal-cardinal quality structure. The structure is built from two relations on quality vectors: $R_1$ encodes 'at least as good on every metric,' and $R_2$ encodes preference-intensity comparisons on the cardinal metric given ordinal bounds. The test statistic is $D(S,S') = \\inf_{u \\in \\mathcal{U}_{\\mathcal{P}}} \\left( \\mathbb{E}_{\\pi} u(\\Phi(S,\\cdot)) - \\mathbb{E}_{\\pi} u(\\Phi(S',\\cdot)) \\right)$, the smallest expected-utility advantage of $S$ over $S'$ among all compatible utilities, and its empirical version is fed into a permutation test; the front test rejects $H_0$ if and only if every pairwise subtest rejects. This machinery is what lets the analysis mix Q*Text (cardinal) with two 5-point Likert ratings (ordinal) and quantify robustness to i.i.d. violations via contamination-sized p-value curves.","core_discovery":"On the paper's own terms, the discovery is that a decoding strategy can be admitted to or excluded from a GSD-front by statistical test, and that human-written completions pass that test against current automatic strategies. Concretely, the paper rejects $H_0: H \\notin \\mathrm{gsd}(\\{H, BS, CS, TS, Tk, Tp\\})$ at $\\alpha = 0.05$ because all five pairwise GSD-comparison tests reject their nulls; it therefore concludes that human completion is not significantly outperformed by any automatic decoding strategy in the set and can be assumed to lie in the GSD-front. The paper presents this not as proof that humans write better text, but as evidence that human text is not redundant: there is no automatic strategy that is at least as good on every considered quality dimension for every utility representation compatible with the data. The same test, the paper argues, lets a researcher decide whether a newly proposed decoding strategy is competitive before investing further in it.","pith_inferences":["A natural follow-up not pursued here: rerun the same GSD-front test with independent, non-author annotators; the paper's human-superiority result would be on firmer ground if the front membership of H survives that replacement.","An editorial inference: the requirement that all pairwise subtests reject makes front membership a conservative claim, so a strategy that fails the test is clearly dominated, while one that passes is merely not dominated, not necessarily best.","If benchmark suites adopt contamination bounds as standard reporting, comparisons across papers become more meaningful because readers learn how fragile each ranking is.","The framework could be turned around to compare decoding strategies against each other without a human reference, producing a partial-order answer to which decoder to use."],"forward_implications":["Any new decoding strategy can be certified as competitive only if it survives the GSD-front test against the existing set, replacing single-metric leaderboards with a statistical partial order.","Benchmark papers can report not just point estimates but a significance level and a contamination budget, the number of non-i.i.d. prompts that would overturn the verdict.","Human and automatic evaluations can be analyzed jointly without forcing ordinal scales into cardinal numbers, so no arbitrary weighting scheme is needed.","The framework is transferable to other generation tasks, such as summarization and translation, where multiple quality dimensions compete.","For the specific comparison studied, the result implies current decoding strategies have not yet reached a quality level that makes human-written completions redundant."],"supporting_citations":[{"why":"Supplies the GSD-front definition, the permutation test for S in gsd(S), and the contamination-based robustness analysis used for all statistical decisions.","marker":"Jansen et al., 2024"},{"why":"Provides Q*Text, the cardinal automatic metric combining perplexity, diversity, and coherence, plus the earlier multicriteria framework this paper extends.","marker":"Garces-Arias et al., 2025b"},{"why":"Provides the WikiText prompt source used to build the evaluation sample.","marker":"Merity et al., 2016"},{"why":"Provides the WikiNews benchmark split used alongside WikiText for the 50 prompts.","marker":"Li et al., 2023"},{"why":"Supplies the Qwen2.5-7B model that generates the automatic completions.","marker":"Yang et al., 2024"},{"why":"Defines contrastive search, one of the five decoding strategies, and the coherence metric used in Q*Text.","marker":"Su et al., 2022"},{"why":"Defines nucleus (top-p) sampling, a compared strategy, and documents the degeneration problem that motivates diversity as a quality dimension.","marker":"Holtzman et al., 2019"},{"why":"Defines top-k sampling, one of the five compared decoding strategies.","marker":"Fan et al., 2018"},{"why":"Defines beam search, one of the five compared decoding strategies.","marker":"Freitag & Al-Onaizan, 2017"},{"why":"Supports the blind evaluation protocol used to collect the ordinal human ratings from the two author evaluators.","marker":"Belz et al., 2020"}],"fun_headline_variants":["Human text still undefeated by five decoders","No LLM decoder dominates human writing","Human completions hold the GSD front vs five decoders","Statistical test finds no decoder beats human text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human quality ratings used as the two ordinal dimensions come from the paper's own authors, and the cardinal metric Q*Text was tuned in earlier work to match human ratings from the same research line; if those ratings already favor human-written text, the conclusion that human text is not dominated is largely built into the data.","fun_headline_variants_meta":{"raw":{"variants":["Human text still undefeated by five decoders","No LLM decoder dominates human writing","Human completions hold the GSD front vs five decoders","Statistical test finds no decoder beats human text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2543,"prompt_tokens":912,"completion_tokens":1631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1571}},"tokens_in":528,"tokens_out":1631,"duration_ms":11600,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:55:10.986684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the two author Likert ratings with ratings of the same 300 continuations by independent, non-author annotators who are blind to source, and rerun the five pairwise GSD tests; if for any decoding strategy the test then fails to reject dominance over human text at $\\alpha = 0.05$, the paper's central claim about human text lying in the GSD-front is refuted.","supporting_citations":[{"cited_title":"Hierarchical neural story generation, 2018","cited_arxiv_id":null,"evidence_quote":"Defines top-k sampling, one of the five compared decoding strategies."}],"review_version":2}