Pith. sign in

REVIEW 3 major objections 5 minor 19 references

BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BenCzechMark introduces a 50-task, 8-category Czech benchmark that ranks open-weight language models by the proportion of statistically significant pairwise wins ('duels') per task.

desk verdict A genuinely useful Czech benchmark artifact with an honest but underemphasized multiple-comparisons problem in its headline scoring; deserves a real referee. read the letter →

arxiv 2412.17933 v2 pith:WJ2RKY2J submitted 2024-12-23 cs.CL cs.AI

Martin Fajcik , Martin Docekal , Jan Dolezal , Karel Ondrej , Karel Beneš , Jan Kapsa , Pavel Smrz , Alexander Polok
show 10 more authors
Michal Hradis Zuzana Neverilova Ales Horak Radoslav Sabol Michal Stefanik Adam Jirkovsky David Adamczyk Petr Hyner Jan Hula Hynek Kydlicek
This is my paper · ORCID
classification cs.CLcs.AI
keywords CzechlanguagebenchmarkduelscoringstatisticalsignificancetestingsocialchoiceaggregationBordacountLLMevaluationcontaminationanalysisCzech-centricmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BenCzechMark (BCM) is presented as the first comprehensive Czech-language benchmark for evaluating large language models (LLMs). It combines 50 tasks spanning 8 categories, with 14 newly collected native-Czech datasets, and ranks models not by averaging accuracy but by a 'duel' mechanism: for each task, every pair of models is tested for a statistically significant improvement, and a model's Duel Win Score is the share of duels it wins. Task scores are aggregated into category and overall scores using a Borda-count-style social-choice rule that handles ties. The paper also contributes BUT-LCC, a cleaned 320GB Czech corpus used for contamination filtering and for training the first Czech-centric 7B model, and a leaderboard of 50 model submissions. The authors argue this design addresses three limitations of prior benchmarks: narrow task coverage, miscalibrated classification metrics, and fragile accuracy averaging.

What carries the argument

The load-bearing object is the 'duel' — a pairwise significance test between two models on a single task. For accuracy and exact-match metrics the test is a one-tailed paired t-test; for AUROC it is a Bayesian significance test with Monte-Carlo integration; for perplexity it is a bootstrap. The Duel Win Score (DWS) is the proportion of duels a model wins significantly at α=0.05, Category Win Score (CWS) averages DWS over tasks in a category, and Overall Duel Win Score (OWS) averages CWS across the 8 categories, with ties handled by a Borda-count-style rule in which each task acts as a voter and models as candidates. This mechanism is what the paper claims makes the ranking resilient to chance improvements and to changes in the set of evaluated models.

What would settle it

Split each task's test set into two halves, recompute the Duel Win Score per half, and measure the rank correlation of model orderings between halves; if Kendall τ falls well below 0.8 for a substantial fraction of tasks, the task-level scores are not reproducible and the benchmark cannot support fine-grained comparison.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that BenCzechMark is the first comprehensive multitask, multimetric benchmark for open-weight LLMs in Czech, and that its duel-scoring aggregation makes model comparison fairer than standard averaging. For every task and every pair of models, a one-tailed significance test (a paired t-test for accuracy and exact match, a Bayesian test for AUROC, and bootstrapping for perplexity) decides a win, loss, or tie at α=0.05. A model's Duel Win Score is the proportion of duels it wins significantly; these scores are averaged within task categories (Category Win Score) and then across categories (Overall Duel Win Score) using a Borda-count-inspired rule with ties. The paper reports that with 50 submitted models the resulting leaderboard is stable under removal of up to 20 models (Kendall τ typically high for multi-task categories), identifies a contaminated model family (Qwen2.5) via a canary string, and finds that Czech-specific tokenization gives their 7B model an advantage in language modeling perplexity but not in understanding tasks. The authors also propose max-centered variance (MCV) as a per-model and per-task measure of prompt sensitivity.

Load-bearing premise

The load-bearing premise is that testing each pair of models separately at α=0.05, with no correction for the thousands of pairwise duels, is enough to keep chance improvements from distorting task-level Duel Win Scores.

Editorial extensions

If this is right

  • A new open-weight model can be submitted to the BCM leaderboard and compared across 50 Czech tasks, with a stability-tested Overall Duel Win Score that does not rely on a single average metric.
  • Czech-specific progress can be located by category (math, NLI, sentiment, NER, reading comprehension, language modeling, factual knowledge, Czech language understanding), so a model's overall win can be separated from its failure on specific task families such as Umime-to-Czech.
  • The prompt-sensitivity measure (MCV) tells users which models and tasks need multi-prompt evaluation, preventing one lucky prompt from being mistaken for capability.
  • Because the benchmark only uses open-weight models (log-likelihood access), the leaderboard provides a reproducible comparison point for the open model ecosystem rather than for API-based systems.
  • The release of BUT-LCC and the Czech-centric models gives other teams a cleaned corpus and baseline for continuous pretraining in Czech.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same duel-scoring machinery could transfer to other under-resourced languages, where a few dozen native tasks plus significance-based aggregation may yield fairer rankings than translated MMLU clones.
  • The paper's own limitation data (72% more duels become non-significant under Benjamini-Hochberg correction, while category rankings stay moderately correlated) suggests that consumers should trust coarse category-level rankings over individual task-level DWS values.
  • A direct test of the stability claim would be to recompute the leaderboard after removing the historically weakest models; if the top-of-table ordering changes, the 'stable comparison' claim holds only for the full set of 50 submissions, not for smaller evaluation runs.
  • The superhuman model performance on the poorly-translated HellaSwag-CZ (humans ~60%, best model >70%) suggests translated benchmarks can measure translation artifacts, so future Czech benchmarks should treat automatically translated tasks with caution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BenCzechMark, a Czech-language multitask benchmark with 50 tasks across 8 categories, a leaderboard with 50 model submissions, a duel scoring mechanism (DWS) based on pairwise statistical significance tests, and a category/overall win score aggregation inspired by Borda count. It also contributes BUT-LCC, a large cleaned Czech corpus, a contamination analysis pipeline, and several Czech-centric baseline models. The headline methodological claim is that the duel scoring system is 'grounded in statistical significance theory' and that it mitigates chance-driven improvements when comparing models.

Significance. If the methodological claims hold, BenCzechMark would be a valuable resource for Czech LLM evaluation: it is notably larger in task coverage than prior Czech benchmarks, it ships a maintained leaderboard, it includes newly collected native Czech tasks, it performs systematic contamination checks against a large corpus, and it releases the corpus and baselines. The work also proposes a concrete aggregation mechanism (DWS with Borda-style averaging) that goes beyond simple metric averaging. The paper is transparent about its own limitations, including the multiple-comparison issue discussed in Section 9, which strengthens the overall credibility of the artifact even though it weakens the central claim.

major comments (3)
  1. [Section 4.3 and Section 9] The central claim that DWS is 'grounded in statistical significance theory' and mitigates chance improvements is not supported at task granularity. Section 4.3 applies a one-tailed paired t-test at α=0.05 per duel without multiple-comparison control. With N=50 models, each task involves 1,225 pairwise duels, so under the global null roughly 61 false-positive duels per task are expected. The authors' own Section 9 analysis shows that Benjamini-Hochberg correction makes 72% more duels non-significant and, for three tasks, reduces non-zero DWS values to less than half. Since the default leaderboard and Figures 4 and 8 use the uncorrected per-duel procedure, task-level DWS values and per-task conclusions drawn from the default leaderboard are not reliable as significance-based measures. I recommend using the DWS-level corrected procedure as the primary scoring, or at least reporting all task-level analyses with corrected DWS and restricting the 'significance-grounded' claim to rank-level stability.
  2. [Section 7, 'Model Contamination'] The text states that Qwen2.5 models are excluded from evaluation, but that 'Reported DWS still captures duels with these models.' If those models are excluded from the leaderboard, their duels should not contribute to the denominator of other models' DWS values; if they are retained in the DWS computation, then the exclusion has no effect on scores and the contamination-based exclusion is purely cosmetic. Please clarify how DWS was recomputed after removing Qwen2.5, and if the reported DWS values include duels with excluded models, the leaderboard and figures should be regenerated.
  3. [Section 9, last paragraph] The distinction between 'duel-level' (A) and 'DWS-level' (B) statistical guarantees is important, but the paper does not state which analyses in Section 6 use which guarantee. The default leaderboard is stated to use (A), and the headline OWS/CWS results in Figures 2 and 4 appear to be based on (A). If the leaderboard allows switching to (B), the paper should specify which of the reported results are affected by the switch, and it should not present (A)-based task-level values as statistically robust.
minor comments (5)
  1. [Abstract and Introduction] The phrase 'the first comprehensive Czech language benchmark' is contestable given the concurrent CzechBench (Section 2 explicitly acknowledges it); consider softening to 'the largest' or 'the most comprehensive at time of writing.'
  2. [Figure 5] The hierarchical clustering dendrogram is difficult to read in its current rendering; the task labels overlap and the caption does not describe the distance metric or linkage criterion beyond 'cosine similarity' and 'hierarchical clustering.' Please make the figure legible and specify the clustering details.
  3. [Section 4.4, Equation (1)] The max-centered variance V_m(P_t) divides by |P_t|−1 even when a single prompt is used; clarify the handling of tasks with one prompt (e.g., LMF tasks) and state the averaging procedure over tasks explicitly.
  4. [Section 7, Table 2] The table lists several datasets with contamination above 50% (e.g., SumeCzech, CzechNews) that were removed, while other high-contamination datasets (e.g., Propaganda, Czech Court Decisions) were retained with a justification based on label leakage. The text is clear, but a short sentence summarizing the retention criterion in the main text would help readers who do not see the appendix.
  5. [Section 9, first paragraph] The limitation that BCM supports only open-weight models is stated, but the consequence for the leaderboard's representativeness (no closed API models) could be discussed more; it is a design choice that also excludes models otherwise central to current LLM deployment.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: BenCzechMark's DWS/CWS/OWS and MCV are computed from public test-set outputs via a transparent, externally reproducible pipeline; the significance-testing and Borda-style aggregation steps do not reduce to their inputs, and the self-citations present are provenance/related-work references, not load-bearing derivations.

full rationale

I walked the derivation chain of the paper's central claims: Duel Win Score, Category Win Score, Overall Win Score, and Max-Centered Variance. Each is computed from model outputs on public test sets using documented formulas: DWS is the proportion of pairwise duels where a one-tailed paired t-test (or Bayesian AUROC test, or bootstrapping for perplexity) finds significant improvement at alpha=0.05; CWS and OWS are averages of DWS over tasks and categories; MCV is a descriptive statistic over prompt scores. No parameter is fitted to a target and then renamed a prediction, and no equation defines one claimed output in terms of another claimed output. The BH-correction discussion in Section 9 is a statistical-power and validity limitation, not a circular step: retaining the less stringent duel-level guarantee does not make DWS equivalent to its inputs, it merely weakens the false-discovery guarantee. The authors explicitly acknowledge that FDR control would make 72% more duels non-significant and that three tasks would see non-zero DWS drop by more than half; this is honest reporting of a methodological trade-off, not a reduction of the result to its premise. The paper's self-citations (CzechBench/Jirkovský et al., Štefánik et al., Kydlíček et al., Horák et al., Fajcik et al.) are used for dataset provenance, related-work contrast, and standard-metric justification; none is invoked as an external theorem that forces the benchmark's ranking mechanism. The contamination analysis and canary-string check are data-hygiene procedures rather than derivations; the concern that the canary string is identical to BIG-bench's is a correctness issue about the contamination inference, not circularity. The benchmark's own internal stability analysis (Kendall tau under model removal) is empirical robustness evidence, not a circular validation. Therefore no load-bearing step reduces to its own inputs, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The benchmark rests on measurement choices, including significance level, best-prompt selection, and contamination detection, rather than fitted physical parameters. The key invented constructs are the DWS and MCV metrics, both publicly reproducible.

free parameters (3)
  • duel significance level alpha = 0.05
    Chosen a priori for all pairwise significance tests; not fitted to data, but directly controls which duels count as wins, so it shapes every DWS.
  • contamination n-gram size = 13 tokens
    Determines which dataset samples are flagged as contaminated in Table 2; affects dataset inclusion and model exclusion decisions.
  • contamination removal threshold = 50% (applied selectively)
    The paper considered removing datasets with at least 50 percent contaminated samples but kept several such datasets, such as Czech Court Decisions at 95.10 percent, based on label-leakage judgment, making the rule post hoc.
assumptions (4)
  • domain assumption A statistically significant per-duel win at alpha=0.05 is a meaningful unit of model comparison, without controlling the false discovery rate across duels.
    Invoked in Section 4.3 for DWS and retained as primary in Section 9 over the stricter Benjamini-Hochberg procedure; the authors report FDR increases non-significant cases by 72 percent.
  • domain assumption AUROC computed one-vs-all and macro-averaged is a threshold-free, fair metric for classification-format tasks.
    Used for CF tasks in Section 4.1 and Appendix B.8; assumes model score ranking across classes is comparable and threshold-free.
  • domain assumption Exact 13-gram match in BUT-LCC is a valid contamination detector for evaluation datasets.
    Used in Section 7 and Table 2 to judge dataset contamination; no precision and recall analysis of the detector is provided.
  • domain assumption The best-performing prompt per model gives a fair estimate of that model's capability.
    In Section 4.3 each model's DWS is computed using its best prompt; the paper measures prompt sensitivity with MCV but still uses the maximum, which can overestimate capability.
invented entities (2)
  • Duel Win Score (DWS) independent evidence
    purpose: Task-level model ranking as proportion of statistically significant pairwise wins; the core of the benchmark's scoring.
    Computed from public per-task model outputs; reproducible by anyone running the released harness and submission scripts.
  • Max-centered variance (MCV) independent evidence
    purpose: Quantifies model and task sensitivity to prompt choice.
    Defined in Equation (1) from per-prompt metric values; recomputable from the released evaluation setup.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism." pith.science (2026). https://pith.science/paper/WJ2RKY2J

@misc{pith2026241217933,
  author       = {Pith},
  title        = {Pith review of: BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WJ2RKY2J}},
  note         = {Machine review of arXiv:2412.17933}
}
read the original abstract

We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its duel scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 14 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis and (ii) continuous pretraining of the first Czech-centric 7B language model with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard with existing 50 model submissions, where new model submissions can be made at https://huggingface.co/spaces/CZLC/BenCzechMark.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 11 canonical work pages

  1. [6]

    arXiv preprint arXiv:2406.08446

    Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08446. Ivan Habernal and Tomáš Brychcín. 2013. Unsu- pervised improving of sentiment analysis using global target context. In Proceedings of RANLP

    arXiv 2013

  2. [7]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt

    Association for Computational Linguis- tics. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Interna- tional Conference on Learning Representations. Zdeˇnka Hladká. 2006. Ksk-dopisy (kor- pus soukromé korespondence): p ˇrepisy ru ˇcnˇe psaný...

    work page 2021

  3. [9]

    In Annual Meeting of the Association for Computational Linguistics

    Truthfulqa: Measuring how models mimic human falsehoods. In Annual Meeting of the Association for Computational Linguistics. Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, Väinö Hatan- pää, Peter Sarlin, and Sampo Pyysalo. 2024. Poro 34b and the blessing of multilinguality. arXiv preprint arXiv:2404.01856. Alex Mallen, Akar...

    arXiv 2024

  4. [10]

    Transactions of the Association for Computational Linguistics, 12:933–949

    State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933–949. Antonio Morais. 2023. A bayesian approach to machine learning model comparison. Idiap- Com Idiap-Com-01-2023, Idiap. Daniel Müllner. 2011. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109....

    arXiv 2023

  5. [12]

    Ústav ˇCeského národního korpusu FF UK, Praha

    2021. Ústav ˇCeského národního korpusu FF UK, Praha. Released corpus. Cyril Goutte and Eric Gaussier. 2005. A proba- bilistic interpretation of precision, recall and f- score, with implication for evaluation. In Euro- pean conference on information retrieval, pages 345–359. Springer. Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and H...

    work page 2021

  6. [13]

    In The Eleventh Interna- tional Conference on Learning Representations

    Language models are multilingual chain- of-thought reasoners. In The Eleventh Interna- tional Conference on Learning Representations. Silma Team. 2024. Silma. Guijin Son, Hanwool Lee, Sungdong Kim, Se- ungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understa...

    arXiv 2024

  7. [14]

    Transactions on Machine Learning Re- search

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Re- search. Michal Štefánik, Marek Kadl ˇcík, Piotr Gramacki, and Petr Sojka. 2023. Resources and few-shot learners for in-context learning in Slavic lan- guages. In Proceedings of the 9th Workshop on Slavic Natural Language Proce...

    work page 2023

  8. [15]

    I don’t know

    Turkishmmlu: Measuring massive mul- titask language understanding in turkish. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4791–4800, Florence, Italy. Association for Comp...

    work page 2023

Show all 19 references
  1. [16]

    5 texts are original Czech works, and 3 are translations from German and Slovak

    consists of 8 literary works published in Czechia between 1991 and 2007, includ- ing six fiction titles, a children’s book, and a memoir (one originally published in 1969). 5 texts are original Czech works, and 3 are translations from German and Slovak. • Karel Havlicek News (...

    1991

  2. [17]

    • Private Correspondence Corpus (Hladká,

    contains all journalistic text written by Karel Havlíˇcek (often referred to as founder of modern Czech journalism) published in his periodicals Pražské noviny (Prague Newspa- per, 1846—1848), including its supplement ˇCeská v ˇcela (The Czech Bee), and Národní noviny (Nationa...

  3. [1901]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al

    Curran Associates, Inc. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-...

    2024 arXiv

  4. [2006]

    <initial 20 ws tokens>, ..., <sampled window>

    (collected between 1990-2004) allows insight into the language and style of con- temporary epistolary texts of a private nature. This corpus captures what might be the final stage in the existence of traditional handwrit- ten correspondence. • Spoken (originally ORAL2013 (Bene...

    2023

  5. [2013]

    Ústav ˇCeského národ- ního korpusu FF UK, Praha

    Oral2013: Reprezentativní korpus nefor- mální mluvené ˇceštiny. Ústav ˇCeského národ- ního korpusu FF UK, Praha. Yoav Benjamini and Yosef Hochberg. 1995. Con- trolling the false discovery rate: A practical and powerful approach to multiple testing. Jour- nal of the Royal Stati...

    1995

  6. [2015]

    The speeches were made by Czech presidents, or their deputies, announced periodically on the occasion of anniversaries and public holidays (New Year, 28 October, etc.)

    is a corpus of presidential speeches from the years 1918-2015. The speeches were made by Czech presidents, or their deputies, announced periodically on the occasion of anniversaries and public holidays (New Year, 28 October, etc.). Speeches is a small and specialized corpus of...

    1918

  7. [2017]

    Ústav ˇCeského národního korpusu FF UK, Praha

    Fictree: Manuáln ˇe syntakticky anoto- vaný korpus ˇceské beletrie. Ústav ˇCeského národního korpusu FF UK, Praha. Available from WWW: http://www.korpus.cz. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Flor...

    2023 arXiv

  8. [2018]

    In BlackboxNLP@EMNLP

    Glue: A multi-task benchmark and anal- ysis platform for natural language understand- ing. In BlackboxNLP@EMNLP. Yequan Wang, Jiawen Deng, Aixin Sun, and Xuy- ing Meng. 2022. Perplexity from plm is unreli- able for evaluating text quality. arXiv preprint arXiv:2210.05892. Coli...

    2022 arXiv

  9. [2021]

    Dialekt: Ná ˇreˇcní korpus, verze 2 z 23

  10. [2023]

    Yixin Nie, Haonan Chen, and Mohit Bansal

    Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In Pro- ceedings of the AAAI conference on art...

    2019 arXiv

  11. [2024]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 7421–7454, Bangkok, Thailand

    MT-bench-101: A fine-grained bench- mark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 7421–7454, Bangkok, Thailand. Association for Comp...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.