Pith. sign in

REVIEW 4 major objections 6 minor 21 references

TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A continuous Elo rating system ranks GPT-4o as the best of 24 models on multilingual toxicity classification.

desk verdict Useful benchmark infrastructure for social-science LLM evaluation, but the first snapshot overclaims: the Perspective claim is wrong and the cross-language ranking is not yet supported by the data. read the letter →

arxiv 2412.00539 v2 pith:QNA3Z5KA submitted 2024-11-30 cs.CL cs.AI

Bastián González-Bustamante This is my paper · ORCID
classification cs.CLcs.AI
keywords largelanguagemodelsEloratingtextclassificationtoxicitydetectionmultilingualevaluationbenchmarkingMeta-Elocomputationalsocialscience
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a continuous benchmarking protocol, TextClass Benchmark, that rates large language models on text classification tasks using a chess-style Elo system, then presents its first cycle on toxicity detection in four languages. The author argues that Elo ratings updated over repeated cycles give a fair, dynamic comparison of closed and open models, and that a weighted Meta-Elo summarises the domain-specific leaderboards. In the first snapshot, GPT-4o is the top model overall, open models such as Qwen 2.5 are close, and every tested LLM beats the older Perspective API. A strong quadratic fit between Meta-Elo and weighted F1 (R2 = 0.981) supports treating Meta-Elo as a compressed ranking that tracks absolute performance. The results also reveal a large gap by language, with models doing well in English and Russian but far worse in Chinese, a gap the paper attributes to language difficulty.

What carries the argument

The load-bearing mechanism is the adapted Elo rating system: models are paired in round-robin matches, the winner is decided by F1-score if the margin exceeds 0.05, ratings update with K = 40 from a 1,500 baseline, and a Meta-Elo combines the language-specific Elo scores using weights for task complexity, language data scarcity, absolute F1 and number of cycles. The same pairwise logic turns raw accuracy, precision, recall and F1 into comparable relative ratings that can be updated as new models and test sets are added.

What would settle it

Build a Chinese toxicity test set matched to the English set in source type, label balance, annotation guidelines and difficulty, then rerun the same models; if GPT-4o's Chinese F1 rises near its English F1, the reported language hierarchy is an artifact of test-set difficulty rather than model capability.

Watch

Extended reading notes

Core claim

The central claim is that the TextClass Benchmark's first cycle provides a fair, cross-lingual snapshot of 24 models on binary toxicity classification, with GPT-4o (2024-11-20) as the best model across leaderboards. Using zero-shot prompting at temperature zero on balanced samples of 5,000 observations per language, the author reports that models reach average F1 of 0.952 in English, 0.910 in Russian, 0.814 in German and only 0.346 in Chinese; GPT-4o posts the top Chinese score (0.751) and the top Russian score (0.952), while Nous Hermes 2 Mixtral leads English (0.977) and Hermes 3 70B leads German (0.848). All evaluated LLMs outperform the Perspective API. The paper further claims that Meta-Elo, a weighted combination of language-specific Elo scores, is a valid compressed measure of relative performance because it relates to weighted F1 through a quadratic curve with R2 = 0.981.

Load-bearing premise

The load-bearing assumption is that the four balanced test sets are comparable in difficulty, so that cross-language differences in F1 reflect model language ability rather than dataset-specific properties.

Editorial extensions

If this is right

  • A researcher selecting a model for incivility classification in these four languages gets a clear default: GPT-4o currently leads the combined Meta-Elo ranking.
  • Meta-Elo can be read as a single number that tracks absolute performance, so future cycles can compare models added at different times on a common scale.
  • The wide gap between English/Russian and Chinese performance means a model's multilingual claims should be checked per language before deployment.
  • Because every tested LLM beat Perspective API, the benchmark suggests classical toxicity-classification tools have been overtaken for this zero-shot task.
  • The continuous design will make model drift and obsolescence measurable: a model's rating now has a temporal trajectory, not just a one-off accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's four test sets come from different source corpora, so the headline language gap may partly reflect dataset difficulty rather than model ability; an equivalent-difficulty test set in Chinese would settle this.
  • The hand-set language weights (Chinese 1.3, Russian 1.4, German 1.1) embed a judgment about scarcity and complexity; changing those weights could reorder the Meta-Elo ranking.
  • The same Elo-plus-Meta-Elo protocol could be transplanted to other social-science classification tasks such as policy agenda coding or misinformation detection, giving a general-purpose leaderboard for the discipline.
  • If later cycles show the quadratic Meta-Elo versus F1 relationship degrading, that divergence would itself be diagnostic of when relative Elo ratings stop reflecting absolute performance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes TextClass Benchmark, an ongoing Elo-based benchmark for LLMs and BERT-family models on text classification tasks relevant to social sciences. It introduces a tailored Elo rating system based on pairwise F1 comparisons and a Meta-Elo aggregation that weights task complexity, language data scarcity, normalized absolute performance, and cycle count. The first cycle covers toxicity/incivility classification in Chinese, English, German, and Russian for 24 models with 96 evaluations, reporting per-language accuracy, precision, recall, F1, Elo, and Meta-Elo. The headline results are a language hierarchy (English and Russian highest, Chinese markedly lowest), GPT-4o as the best model in Meta-Elo, and a reported quadratic relationship between weighted F1 and Meta-Elo with R2 = 0.981.

Significance. The infrastructure is timely and potentially useful: a transparent, continuously updated leaderboard with zero-temperature settings, fixed test sets, and public code and data could help social scientists choose models for text annotation. The explicit formulas for Elo and Meta-Elo make the method reproducible. However, the current snapshot's central comparative claims are not yet supported: the cross-lingual test sets are not shown to be comparable, the reported R2 = 0.981 is partly an artifact of F1 appearing on both sides of the relationship, and one stated finding is contradicted by the paper's own Figure 1. These issues are substantial but fixable with additional analysis and careful reframing.

major comments (4)
  1. [Section 4, Figure 1] Section 4 states that 'another interesting finding is that all LLMs outperform more classical transformer approaches such as Perspective API,' but Figure 1 shows Perspective 0.80 (ZH) and Perspective 0.80 (DE) placed above GPT-4o (ZH, F1 = 0.751) and above many open-weight LLMs in the Chinese and German blocks. Because Perspective API is included in the leaderboard and outperforms the flagship LLM in Chinese and German, the sentence as written is false. Please restrict the claim to the languages where it holds or report per-language comparisons; this matters because the abstract and introduction position the benchmark as demonstrating LLM superiority over older tools.
  2. [Section 4, Figure 2; Equations (5)-(6)] The high R2 between weighted F1 and Meta-Elo is expected by construction. Equation (6) defines the Meta-Elo weight w_j to include w_F1, the normalized F1-Score, and the x-axis 'weighted F1-Score' is computed by 'emulating the abovementioned process' (Section 4). Thus F1 contributes to both variables, so R2 = 0.981 cannot be read as independent validation that Meta-Elo summarizes absolute performance. Report the correlation between F1 and a version of Meta-Elo computed without the w_F1 weight, or with out-of-sample weights, and avoid presenting the quadratic fit as evidence of validity.
  3. [Section 3 and Section 2.2] The four test sets are drawn from different source corpora (e.g., Jigsaw/Unitary for English, DeTox and GemEval for German; Chinese and Russian sources are not cited) and are described only as a 'balanced sample of 5,000 observations per country.' The paper then interprets average F1 differences (0.952 English vs. 0.346 Chinese) as a 'hierarchy by language' and uses hand-set language weights (Section 2.2) that presuppose that Chinese and Russian are intrinsically harder. Without evidence that the sets are equated in label distribution, annotation quality, and item difficulty, the cross-language ranking is not well-defined. At minimum, provide per-language label balance, validation statistics, and dataset descriptions; otherwise present each language as a separate leaderboard and drop the cross-lingual hierarchy claim.
  4. [Section 2.1-2.2] The ranking depends on several free parameters (K = 40, draw threshold 0.05, language weights 1.3/1.1/1.4, cycle weight 1 + log(cycle + 1)) whose values are asserted without justification or sensitivity analysis. Since the top Meta-Elo position is driven by GPT-4o's Chinese and Russian F1, it is important to show that the conclusion is robust to reasonable alternatives, e.g., K = 20 or 60, draw thresholds of 0.03/0.07, and equal language weights. The paper should also report uncertainty (e.g., bootstrap confidence intervals for F1 and Elo), as many models are separated by less than the 0.05 draw threshold.
minor comments (6)
  1. [Section 5] The heading 'A venues' should be 'Avenues,' and the text uses 'wights' where 'weights' is intended.
  2. [Section 3] The text says '5,000 observations per country,' but the data are per language; use 'per language' to avoid confusion.
  3. [Equation (5)] The summation symbol in Equation (5) is typeset as 'nX'; please replace with proper LaTeX summation notation.
  4. [Section 3] The Chinese and Russian data sources are not identified; provide citations or URLs for these subsets, as the reproducibility of the benchmark depends on them.
  5. [Figure 1] The Perspective API entries are labeled with thresholds (0.80, 0.70, 0.60, 0.55) that are not explained in the text or caption; define what these numbers represent.
  6. [Footnote 1] The planned weights for Arabic, Hindi, and Spanish are listed but not used in this cycle; clarify that these are prospective values rather than results from the current snapshot.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Elo and Meta-Elo are computed from raw F1 scores, and the Figure 2 correlation is a consistency check, not a fitted prediction.

full rationale

The derivation chain is self-contained: pairwise F1 comparisons determine match outcomes (Section 2.1), ratings update via the standard Elo equations, and Meta-Elo is a weighted sum of those ratings (Eqs. 5-6). No parameter is fitted to a subset of the leaderboard data and then relabeled as a prediction of those same data. The Figure 2 regression of weighted F1 on Meta-Elo (R2 = 0.981) is expected because Elo scores are functions of F1, but the paper presents it as an illustration of consistency ('strong relationship between absolute and relative performance'), not as an independent result or a derivation. The language weights are hand-set a priori and are not fitted from outcomes, so they do not make the ranking circular. The single self-citation for the prompt (González-Bustamante, 2024) is not load-bearing because the prompt is quoted verbatim in Section 3. Cross-language dataset comparability is a validity concern for the 'hierarchy by language' claim, but it is not a circularity of the paper's formal derivation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or theoretical entities are introduced. The free parameters are all methodology choices that directly shape the reported rankings and Meta-Elo values. The axioms are the Elo model, the F1-based outcome rule, the draw threshold, the hand-set weights, and the stability assumption. No entity has independent falsifiable evidence outside the benchmark itself.

free parameters (6)
  • language weight for Chinese = 1.3
    Hand-set in Section 2.2 to reflect data scarcity and linguistic complexity; directly scales Chinese leaderboard contributions to Meta-Elo.
  • language weight for German = 1.1
    Hand-set in Section 2.2.
  • language weight for Russian = 1.4
    Hand-set in Section 2.2.
  • Elo K-factor = 40
    Chosen for quick adjustments (Section 2.1); a smaller K would slow rating changes.
  • F1 draw threshold = 0.05
    Margin defining win versus draw (Section 2.1); no statistical justification.
  • cycle weight = 1 + log(cycle + 1)
    Formula in Section 2.2 to reward consistently benchmarked models; no empirical basis.
assumptions (5)
  • standard math Elo expected-score formula (Equations 1-2)
    Standard Elo model borrowed from chess; unproved background used to convert pairwise outcomes into ratings.
  • domain assumption F1-Score is the primary absolute performance metric and determines win/loss/draw
    The paper chooses F1 as the arbiter of model superiority (Section 2.1); alternative metrics could change rankings and draws.
  • ad hoc to paper Draw threshold of 0.05 F1 points
    No statistical justification; with no confidence intervals, a 0.05 margin may be within noise.
  • ad hoc to paper Language weights (Chinese 1.3, German 1.1, Russian 1.4) and cycle weight 1+log(cycle+1) encode data scarcity, linguistic complexity, and obsolescence
    Values are hand-picked in Section 2.2 and directly determine Meta-Elo rankings; no empirical calibration is provided.
  • domain assumption Temperature zero and standard Ollama parameters yield stable, comparable generations across models
    Tooling assumption; the paper itself acknowledges reproducibility issues with LLMs in Section 1, and no repeated-run variance is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences." pith.science (2026). https://pith.science/paper/QNA3Z5KA

@misc{pith2026241200539,
  author       = {Pith},
  title        = {Pith review of: TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNA3Z5KA}},
  note         = {Machine review of arXiv:2412.00539}
}
read the original abstract

The TextClass Benchmark project is an ongoing, continuous benchmarking process that aims to provide a comprehensive, fair, and dynamic evaluation of LLMs and transformers for text classification tasks. This evaluation spans various domains and languages in social sciences disciplines engaged in NLP and text-as-data approach. The leaderboards present performance metrics and relative ranking using a tailored Elo rating system. With each leaderboard cycle, novel models are added, fixed test sets can be replaced for unseen, equivalent data to test generalisation power, ratings are updated, and a Meta-Elo leaderboard combines and weights domain-specific leaderboards. This article presents the rationale and motivation behind the project, explains the Elo rating system in detail, and estimates Meta-Elo across different classification tasks in social science disciplines. We also present a snapshot of the first cycle of classification tasks on incivility data in Chinese, English, German and Russian. This ongoing benchmarking process includes not only additional languages such as Arabic, Hindi, and Spanish but also a classification of policy agenda topics, misinformation, among others.

Figures

Figures reproduced from arXiv: 2412.00539 by the authors.

Figure 1
Figure 1. Goodness-of-Prediction Metrics, Elo-Score, and Meta-Elo [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Weighted F1-Score and Meta-Elo Note. Green: open-source; red: closed-private; violet: BERT family. Models with a weighted F1-Score lower than 0.7 were excluded. Bubble size represents the —presumed for GPTs— number of parameters. approaches such as Perspective API. Elo-Score allows us to visualise better the differ￾ences in languages since it tends to order the per￾formance within the domain-specific leaderboards. M… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Argyle, Ethan C

    Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. 2023. https://doi.org/10.1017/pan.2023.2 Out of One, Many: Using Language Models to Simulate Human Samples . Political Analysis, 31(3):337--351

    doi:10.1017/pan.2023.2 2023

  4. [4]

    Christopher Barrie, Elli Palaiologou, and Petter Törnberg. 2024 a . https://doi.org/10.48550/ARXIV.2407.02039 Prompt Stability Scoring for Text Annotation with Large Language Models . Preprint , arXiv

    arXiv doi:10.48550/arxiv.2407.02039 2024

  5. [5]

    Christopher Barrie, Alexis Palmer, and Arthur Spirling. 2024 b . Replication for Language Models: Problems, Principles, and Best Practice for Political Science . Preprint , APSA

    work page 2024

  6. [6]

    Daryna Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinew Ali Ayele, Naquee Rizwan, Frolian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashraf Elnagar, Animesh Mukherjee, and Alexander Panchenko. 2024. Overview of the Multilingual Text Detoxification Task at PAN 2024 . In Working Notes of CLEF 2024 - Co...

    work page 2024

  7. [7]

    Christoph Demus, Jonas Pitz, Mina Schütz, Nadine Probol, Melanie Siegel, and Dirk Labudde. 2022. https://doi.org/10.18653/v1/2022.woah-1.14 A Comprehensive Dataset for German Offensive Language and Conversation Analysis . In Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pages 143--153. Association for Computational Linguistics

    doi:10.18653/v1/2022.woah-1.14 2022

  8. [8]

    Mingmeng Geng, Sihong He, and Roberto Trotta. 2024. https://doi.org/10.48550/ARXIV.2405.19323 Are Large Language Models Chameleons? Preprint , arXiv

    work page Pith review arXiv doi:10.48550/arxiv.2405.19323 2024

Show all 21 references
  1. [9]

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. https://doi.org/10.1073/pnas.2305016120 ChatGPT outperforms crowd workers for text-annotation tasks . Proceedings of the National Academy of Sciences, 120(30):e2305016120

    2023 doi

  2. [10]

    Bastián González-Bustamante. 2023. https://doi.org/10.5287/ORA-QZ8KDD44G Critical events and ministerial turnover in Latin American presidential democracies . Ph.D. thesis, St Hilda's College, University of Oxford

    2023 doi

  3. [11]

    Bastián González-Bustamante. 2024. https://doi.org/10.48550/ARXIV.2409.09741 Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Data . Preprint , arXiv

    2024 doi arXiv

  4. [12]

    Gruber and Maximilian Weber

    Johannes B. Gruber and Maximilian Weber. 2024. https://doi.org/10.48550/ARXIV.2404.07654 rollama: An R package for using generative large language models through Ollama . Preprint , arXiv

    2024 doi arXiv

  5. [13]

    Laura Hanu and Unitary. 2020. Detoxify . Github. https://github.com/unitaryai/detoxify

    2020

  6. [14]

    Guozhi Hao, Jun Wu, Qianqian Pan, and Rosario Morello. 2024. https://doi.org/10.1038/s41598-024-66708-4 Quantifying the uncertainty of LLM hallucination spreading in complex adaptive social networks . Scientific Reports, 14(1):16375

    2024 doi

  7. [15]

    Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen

    Xingwei He, Zhenghao Lin, Yeyun Gong, A.-L. Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. https://doi.org/10.48550/ARXIV.2303.16854 AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators . Preprint , arXiv

    2024 doi arXiv

  8. [16]

    Michael Alvarez

    Mitchell Linegar, Rafal Kocielnik, and R. Michael Alvarez. 2023. https://doi.org/10.3389/fpos.2023.1257092 Large language models and political science . Frontiers in Political Science, 5:1257092

    2023

  9. [17]

    Julian Risch, Anke Stoll, Lena Wilms, and Michael Wiegand. 2021. Overview of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments . In Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-...

    2021

  10. [18]

    Arthur Spirling. 2023. https://doi.org/10.1038/d41586-023-01295-4 Why open-source generative AI models are an ethical way forward for science . Nature, 616(7957):413--413

    2023 doi

  11. [19]

    Timoneda and Sebastian Vallejo Vera

    Joan C. Timoneda and Sebastian Vallejo Vera. 2024. https://doi.org/10.1086/730737 BERT, RoBERTa or DeBERTa? Comparing Performance Across Transformer Models in Political Science Text . The Journal of Politics. OnlineFirst

    2024 doi

  12. [20]

    Kohei Watanabe and Yuan Zhou. 2022. https://doi.org/10.1177/0894439320907027 Theory-Driven Analysis of Large Corpora: Semisupervised Topic Classification of the UN Speeches . Social Science Computer Review, 40(2):346--366

    2022 doi

  13. [21]

    Maximilian Weber and Merle Reichardt. 2023. https://doi.org/10.48550/ARXIV.2401.00284 Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models . Preprint , arXiv

    2023 doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.