Pith. sign in

REVIEW 4 major objections 6 minor 4 references

A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o reaches the highest overall accuracy on the argument-classification benchmarks tested, while DeepSeek-R1 leads plain accuracy on Args.me; a certainty-weighted voting scheme lifts scores for nearly every model.

desk verdict A useful but statistically under-supported LLM benchmark for argument classification: the headline rankings are plausible, yet the gaps are within one standard error of 200-item subsamples. read the letter →

arxiv 2507.08621 v2 pith:IZY7KYGT submitted 2025-07-11 cs.CL cs.AI

Marcin Pietroń , Rafa{l} Olszowski , Jakub Gomu{l}ka , Filip Gampel , Andrzej Tomski This is my paper · ORCID
classification cs.CLcs.AI
keywords argumentminingclassificationlargelanguagemodelspromptengineeringchain-of-thoughtcertainty-weightedvotingUKPdatasetArgs.mecorpus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show how well modern large language models classify arguments into 'for', 'against', and 'no argument' categories, comparing Llama, GPT-4, GPT-4o, and DeepSeek-R1 on the UKP and Args.me corpora. The authors report that GPT-4o reaches the best overall accuracy, with 84.3% on UKP under certainty-weighted voting and 90.6% on Args.me, while DeepSeek-R1 is the best plain-accuracy model on Args.me at 90.1%. They also claim that their certainty-weighted multi-prompt voting scheme improves accuracy over single-prompt answers. Error analysis shows that models most often mistake neutral statements for arguments, and that a nontrivial share of apparent model errors are actually annotation mistakes. If these results hold, they give practitioners a concrete ranking of off-the-shelf LLMs for argument mining and a simple voting recipe to improve it.

What carries the argument

The load-bearing mechanism is the certainty-weighted multi-prompt voting algorithm (Algorithm 1): each model answers four prompt variants (labeled P1-P4) that differ in thesis phrasing and response format, and when the votes tie, the model's self-reported certainty percentages decide. Prompt variants that demand natural-language answers and simple thesis phrasings consistently do better than letter-code variants, so the voting scheme leans on the stronger prompts while still using all of them. The comparison also uses Chain-of-Thought and few-shot prompting, but the proposed algorithm is the only component that changes scores substantially.

What would settle it

Run the same prompt set on the full UKP and Args.me partitions, or on many fresh 200-entry samples with confidence intervals, and check whether GPT-4o's leads (84.3% vs 80.1% on UKP, 90.6% vs 90.4% on Args.me) persist; also re-adjudicate the disagreement records for both top models on all eight UKP topics to test whether the 28.6% to 41.9% annotator-error share found on minimum wage generalizes.

Watch

Extended reading notes

Core claim

The central claim is that on the two public argument-classification benchmarks, the best-performing off-the-shelf LLMs are GPT-4o and DeepSeek-R1, with GPT-4o ahead overall: 84.3% versus 80.1% mean accuracy on UKP with certainty-weighted voting, and 90.6% versus 90.4% on Args.me with certainty weighting, while DeepSeek-R1 wins plain accuracy on Args.me (90.1% versus 87.2%). The paper also claims that its four-prompt scheme combined with model-reported certainty (Algorithm 1) consistently improves the scores of nearly every model, and that the largest remaining error class is misclassifying neutral statements as arguments. On the UKP minimum-wage partition, manual review of disagreements found that between 28.6% and 41.9% of 'incorrect' answers by two strong models were actually the annotators who were wrong, so the true model accuracy is likely higher than the raw agreement numbers show.

Load-bearing premise

The rankings rest on 200-entry subsamples per dataset partition that are asserted to be statistically representative without specifying the sampling method or computing error bars, so the reported gaps between top models may be sampling noise.

Editorial extensions

If this is right

  • Practitioners can expect GPT-4o to be the strongest default choice for argument classification, with DeepSeek-R1 competitive on plain accuracy for debate-portal arguments.
  • Using a set of four prompt variants with certainty-weighted voting outperforms any single prompt and any three-prompt subset in almost all configurations.
  • Prompt design matters less than model choice: natural-language response formats and simple thesis formulations improve accuracy, while letter-code formats hurt.
  • Chain-of-Thought prompting does not consistently beat the plain prompts, and few-shot examples drawn from other topics can degrade accuracy, especially for smaller Llama models.
  • The common practice of scoring models against dataset annotations understates their true accuracy, since a measurable fraction of disagreements are annotation errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but if the annotator-error rates found in the minimum-wage audit generalize across the datasets, the true ranking between GPT-4o and DeepSeek-R1 could change once both models' 'errors' are re-adjudicated, because the two models disagree with annotations on different records.
  • The finding that smaller Llama models improve more from few-shot examples suggests a testable extension: dynamically retrieving topic-matched examples may close part of the small-model gap, something the paper only gestures at as future work.
  • The certainty-weighted voting scheme is model-agnostic and could be transferred to other classification tasks with three or more conflicting labels, such as stance detection in social media, where tie-breaking by self-reported confidence may be cheaper than adding a verifier model.
  • The paper's 'statistically representative' 200-entry subsamples carry no error bars, so an immediate replication on full partitions or repeated subsampling would tell whether the 1 to 4 percentage point gaps between top models are stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents an empirical comparison of several LLMs (Llama 1B–70B, GPT-4, GPT-4o, DeepSeek-R1, and a limited o3-mini test) on argument classification in the UKP and Args.me datasets. It evaluates four prompt variants, chain-of-thought prompting, few-shot prompting, and a certainty-weighted multi-prompt voting method, and reports accuracy, precision, recall, and F1 for each model. The authors conclude that GPT-4o is the strongest overall classifier, that DeepSeek-R1 is the strongest among reasoning-oriented models on plain accuracy for Args.me, and that the proposed certainty-weighted voting improves results. The paper also includes a qualitative error analysis and a small re-examination of annotator errors.

Significance. If its central rankings are established, the paper would be a useful broad comparison of modern LLMs on argument classification, a relatively underexplored benchmark area, and its qualitative error taxonomy (§5.2) plus the explicit caveats in §5.4 are genuinely informative. The study covers a wide range of model sizes and families, addresses multiple research questions in a structured way, and is among the first to apply GPT-4o and DeepSeek-R1 to these argument corpora. However, the headline claims currently rest on unvalidated 200-item subsamples without statistical error measures, and the proposed voting algorithm is not fully reproducible as written; these issues are load-bearing for the paper's main conclusions.

major comments (4)
  1. [§3, Tables 5–8] The claim that the subsamples are 'statistically representative' is unsupported. Section 3 states that all calculations were performed on subsamples of 200 entries per dataset partition owing to financial constraints, but no sampling protocol, seeds, stratification, or repetition counts are given, and Tables 5–8 report only point estimates with no confidence intervals or significance tests. For a 200-item binomial sample, accuracy near 84% has a standard error of about 2.6 points, so the key reported gaps—GPT-4o 84.3 vs. DeepSeek-R1 80.1 on UKP with certainty weighting, and DeepSeek-R1 90.1 vs. GPT-4o 87.2 on Args.me plain accuracy—are each only about one standard error. These numbers do not by themselves establish the ranking claims. The authors should report confidence intervals or bootstrap estimates, significance tests with multiple-comparison control, and the sampling details, or evaluate on the full datasets.
  2. [Algorithm 1] Algorithm 1 is not reproducible as written. In the non-tie branch (lines 25–27), the variable outl is never assigned, yet line 28 appends outl to A for every model; the 'out' variable computed in the else branch is discarded. In the tie branch, line 23 computes a weighted sum, but the pseudocode does not specify how the products kappa[i]·l(omega) are converted into a final class label. This makes the proposed certainty-weighted voting method (RQ4) underspecified. Please rewrite the algorithm with explicit initialization of outl, an explicit label-selection rule, and a clear description of how certainty values are combined with labels.
  3. [§4.3, §5.1, Algorithm 1] The prompt variants P1–P4, the few-shot examples E1–E3, and the certainty threshold gamma in Algorithm 1 were designed with knowledge of the UKP and Args.me datasets, but the manuscript does not describe how these choices were made or whether any portion of the data was held out for prompt development. In particular, the few-shot examples are topic-specific selections from the eight UKP topics, and gamma is never given a value. This leaves open the possibility that the reported comparisons reflect tuning to the test benchmarks rather than a general comparison of prompting strategies. The authors should specify the selection procedure, report the value of gamma, and ideally evaluate on a held-out split or via cross-validation.
  4. [Abstract, §5.2] The abstract's claim that 'ChatGPT-4o outperforms the others in the argument classification benchmarks' is too broad given the paper's own results. Table 5 shows that on Args.me plain accuracy, DeepSeek-R1 achieves 90.1% versus GPT-4o's 87.2%, and GPT-4o is the best on Args.me only when certainty weighting is used (90.6% vs. 90.4%). The summary claim should be qualified to specify the datasets and settings (e.g., UKP overall, and Args.me with certainty weighting) to avoid presenting a partial result as a global one.
minor comments (6)
  1. [Table 5] The column header 'Deepseek-R1' is spelled inconsistently elsewhere as 'DS R1' and 'DeepSeek-R1'; please standardize the model naming throughout, including Tables 4–10.
  2. [§5.4] The annotator re-analysis is a strength because it honestly discusses the possibility that some model disagreements are annotator errors, and it explicitly warns against overcorrecting; the presentation is clear and appropriately cautious.
  3. [Table 10] Table 10 contains a literal '<empty citation>' entry next to BERT and presents results for 'GPT-4', 'GPT-4o', and 'DS R1' without stating which metrics are directly comparable across rows; please complete the citation and add a note about the evaluation settings used for each baseline.
  4. [Figure 8] The caption for Figure 8 says 'by debate topic in the UKP dataset,' but the surrounding text says it shows error types by model; please correct the caption or the description so the figure and text agree.
  5. [§5.2] The qualitative error examples are useful, but several are anecdotal and not tied to counts or frequencies; adding a small table that shows the frequency of each error type per model would make the claims more precise.
  6. [§4.3 and Appendix A] The few-shot example E2 for cloning contains the word 'Sentence:' inside the quoted example, which appears to be a formatting artifact; please remove or clarify it.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline claims are empirical evaluations against external human-annotated benchmarks; the only self-citation is a non-load-bearing baseline.

full rationale

This paper is an empirical benchmarking study. The central claims—GPT-4o best on UKP with certainty weighting (84.3%), DeepSeek-R1 best on Args.me plain accuracy (90.1%), and certainty-weighted voting improving scores—are all derived from measured agreement with externally human-annotated datasets (UKP and Args.me). No derived quantity is defined in terms of the target result. Equation (4), TΘ(P) = Σ ci·TΘ(pi), and Algorithm 1 are deterministic aggregation rules whose outputs are compared against ground-truth labels provided by independent annotators, not against any model-fitted quantity. The only self-citation, Pietron et al. 2024, appears as a comparison baseline in Table 10 and in related work; it does not support the paper's ranking conclusions, which are computed from fresh LLM runs on the same external benchmarks. The paper's assertion that 200-entry subsamples are 'statistically representative' without sampling details or confidence intervals is a statistical validity concern, not a circularity: it does not make the inputs equivalent to the outputs. Under the required standard of exhibiting a specific reduction from prediction to input, no circular step can be identified, so the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the reliability of human labels and on the sampled subsets; both are assumptions rather than controlled factors. The only explicit tuned constant is the voting threshold, which is unreported.

free parameters (2)
  • Certainty threshold gamma (Algorithm 1) = not reported
    The multiprompt voting algorithm requires a threshold gamma to decide when certainty breaks ties; the value is never given and is a hand-tuned constant.
  • Few-shot example sentences E1-E3 = manually selected per topic
    The few-shot examples in Appendix A were chosen by the authors and shown to sometimes hurt performance, so they act as a manual design parameter.
assumptions (3)
  • domain assumption Human annotation labels in UKP and Args.me are treated as ground truth for computing accuracy.
    Sections 5.1-5.2 compute accuracy as agreement with annotators; Section 5.4 then shows some labels are wrong, partially undermining this assumption.
  • domain assumption The 200-entry subsamples are statistically representative of each dataset partition.
    Section 3 states they are 'statistically representative' without describing the sampling procedure or demonstrating representativeness.
  • ad hoc to paper LLM outputs are stable enough that single responses (not averaged over multiple runs) support the reported percentages.
    The paper does not report repeated runs, temperature, or variance; the tables appear to use one query per item per prompt.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1." pith.science (2026). https://pith.science/paper/IZY7KYGT

@misc{pith2026250708621,
  author       = {Pith},
  title        = {Pith review of: A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IZY7KYGT}},
  note         = {Machine review of arXiv:2507.08621}
}
read the original abstract

Argument mining (AM) is an interdisciplinary research field that integrates insights from logic, philosophy, linguistics, rhetoric, law, psychology, and computer science. It involves the automatic identification and extraction of argumentative components, such as premises and claims, and the detection of relationships between them, such as support, attack, or neutrality. Recently, the field has advanced significantly, especially with the advent of large language models (LLMs), which have enhanced the efficiency of analyzing and extracting argument semantics compared to traditional methods and other deep learning models. There are many benchmarks for testing and verifying the quality of LLM, but there is still a lack of research and results on the operation of these models in publicly available argument classification databases. This paper presents a study of a selection of LLM's, using diverse datasets such as Args.me and UKP. The models tested include versions of GPT, Llama, and DeepSeek, along with reasoning-enhanced variants incorporating the Chain-of-Thoughts algorithm. The results indicate that ChatGPT-4o outperforms the others in the argument classification benchmarks. In case of models incorporated with reasoning capabilities, the Deepseek-R1 shows its superiority. However, despite their superiority, GPT-4o and Deepseek-R1 still make errors. The most common errors are discussed for all models. To our knowledge, the presented work is the first broader analysis of the mentioned datasets using LLM and prompt algorithms. The work also shows some weaknesses of known prompt algorithms in argument analysis, while indicating directions for their improvement. The added value of the work is the in-depth analysis of the available argument datasets and the demonstration of their shortcomings.

Figures

Figures reproduced from arXiv: 2507.08621 by the authors.

Figure 1
Figure 1. System architecture. Each attention layer receives the embeddings. The i-th token is mapped via linear transformations to a key ki , query qi and value vi . The i-th output of the self-attention layer is given by weighting the values vj by the normalized dot product between the query qi and other keys k: zi = Xn j=1 sof tmax({< qi , kj ′ >} n j ′=1)j · vj (3) The TΘ gives an output with the same number of tokens as … view at source ↗
Figure 2
Figure 2. Overview of prompt complexity used in the study. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Heatmap visualizing the performance of the six top-performing LLM models across debate topics in the UKP [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Accuracy distribution of the results for the UKP database per model-topic pair, broken down by prompt types, [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Accuracy distribution of the results for the UKP database per model-topic pair, broken down by prompt types, [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Percentage of each error type depending on the prompt used (sample:UKP dataset). [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Percentage of each error type by debate topic in the UKP dataset. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Percentage of each error type by debate topic in the UKP dataset. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 1 canonical work pages

  1. [1]

    Ajjour, Y ., Wachsmuth, H., Kiesel, J., Potthast, M., Hagen, M., & Stein, B. (2019). Data acquisition for argument search: The args.me corpus. In C. Benzmüller & H. Stuckenschmidt (Eds.), Ki 2019: Advances in artificial intelligence (V ol. 11793). Springer, Cham. https://doi.org/10.1007/978-3-030-30179-8_4 Akiki, C., & Potthast, M. (2020). Exploring argum...

    work page doi:10.1007/978-3-030-30179-8_4 2019

  2. [14]

    https://doi.org/10.1038/s41598-024-79531- 8 19 arXiv Template A PREPRINT Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north American chapter of the association for computat...

    arXiv 2019

  3. [1589]

    University of Richmond

    [Honors Theses]. University of Richmond. https://scholarship.richmond.edu/honors-theses/1589 Daxenberger, J., Eger, S., Habernal, I., Stab, C., & Gurevych, I. (2017). What is the essence of a claim? cross- domain claim identification. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2055–2066. Daxenberger, J., Schill...

    arXiv doi:10.48550/arxiv.2311.04205 2017

  4. [3674]

    I had deep anger when I heard that ,

    Stab, C., & Gurevych, I. (2014). Annotating argument components and relations in persuasive essays. In J. Tsujii & J. Hajic (Eds.), Proceedings of COLING 2014, the 25th international conference on computational linguistics: Technical papers (pp. 1501–1510). Dublin City University; Association for Computational Linguistics. https: //aclanthology.org/C14-11...

    arXiv 2014

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.