Pith. sign in

REVIEW 4 major objections 5 minor 58 references

ArgCMV: An Argument Summarization Benchmark for the LLM-era

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ArgCMV, a benchmark of 12,262 real debate arguments, is harder and more realistic than ArgKP21, and existing key-point extraction models do markedly worse on it.

desk verdict Useful new benchmark for key point extraction in long online debates, but LLM-generated gold labels validated on only 2.5% of data leave the 'harder' claim under-supported. read the letter →

arxiv 2508.19580 v1 pith:5EESGUW6 submitted 2025-08-27 cs.CL

Omkar Gurjar , Agam Goyal , Eshwar Chandrasekharan This is my paper
classification cs.CL
keywords keypointextractionargumentsummarizationbenchmarkdatasetonlinedebater/ChangeMyViewlong-contextLLMevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ArgCMV is a new benchmark for key point extraction from argumentative text, built from 12,262 comments in more than 3,000 real r/ChangeMyView debates rather than the short, expert-written sentences of the standard ArgKP21 dataset. The paper argues that this makes it a more realistic test: CMV arguments are roughly ten times longer, contain rhetorical and testimony statements that are nearly absent from ArgKP21, and frequently express several key points at once. To build the benchmark, the authors use a large language model to turn each side of a debate into candidate key points and a second model to map those key points back onto individual comments, with a small human validation pass. They report that existing key point extraction methods trained on ArgKP21 lose ground on ArgCMV, while open-weights models around 9B parameters perform better out of the box. If the benchmark is sound, it gives the field a more realistic target for summarization tools meant to help moderators, analysts, and readers follow online discussions.

What carries the argument

The load-bearing object is the two-agent annotation pipeline used to generate the benchmark's ground truth. An extraction agent (the GPT-4o-mini language model) reads the full pool of arguments on one side of a thread and returns concise, non-overlapping key points; a mapping agent (the GPT-4o model) then reads each comment individually and outputs the indices of the key points it expresses. This split between extraction and mapping is what turns messy Reddit comments into reusable gold labels, and it also produces the benchmark's signature property—most arguments map to multiple key points, forcing evaluators to pay for partial coverage.

What would settle it

Have independent human annotators write gold key points for a random sample of 300 ArgCMV arguments without seeing the LLM labels, then compare soft-F1 rankings of the same models against human labels versus LLM labels; if the rankings shift substantially or agreement is near chance, ArgCMV's difficulty scores are measuring agreement with the annotator model, not extraction skill.

Watch

Extended reading notes

Core claim

The paper's central claim is that the widely used ArgKP21 benchmark understates the difficulty of real argument summarization, and that a new corpus, ArgCMV, closes that gap by drawing on 12,262 arguments from 3,131 actual online debate threads. By assigning each author's comments to one stance (original poster as 'pro,' every replier as 'con'), the corpus preserves the back-and-forth structure that ArgKP21 strips away. The ground-truth key points are produced by a two-stage LLM pipeline: one model extracts candidate key points from each stance pool, and a second model decides which key points each individual comment expresses; 300 arguments were manually reviewed, with reported precision an

Load-bearing premise

Everything rests on the assumption that the GPT-generated key points are correct and complete enough to serve as gold labels, yet the authors' own validation covered only 2.5% of the data and found that roughly 16% of sampled arguments had at least one human-identified key point the LLM missed, with low annotator agreement (Cohen's Kappa 0.33) on that judgment.

Editorial extensions

If this is right

  • Models and hyperparameters tuned on ArgKP21 will systematically understate their weaknesses when moved to online discussion data; ArgCMV provides a more demanding evaluation surface.
  • Benchmarking on ArgCMV favors long-context handling and multi-point coverage, so future key point systems must generate several key points per argument and match them to specific comments.
  • Small open-weights models at roughly 9B parameters can perform key point extraction out of the box, suggesting that practical moderation and summarization tools need not depend on the largest commercial APIs.
  • The LLM-based extraction-mapping pipeline offers a low-cost recipe for building similar benchmarks for other platforms and time periods, letting the field expand beyond one curated corpus.
  • The released train/dev/test splits with key points and mappings enable direct comparison and reproducible research on the next generation of argument summarization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because stance in ArgCMV is assigned by author role (original poster versus replier), the benchmark cannot test whether a model recognizes when an opponent partially agrees with the other side; a paired-annotation study would reveal whether stance labels blur overlapping reasoning.
  • The reported low inter-annotator agreement on coverage (Kappa 0.33) suggests that 'completeness' of key points is inherently unstable; an immediate stress test is to evaluate generators against independently authored human key points, not only LLM labels.
  • The same extraction-plus-mapping pipeline generalizes readily to other threaded platforms, making cross-platform transfer a natural next experiment; the 2020-only collection window and possible pretraining leakage could be checked by re-running on post-2024 threads.
  • Because 75 percent of ArgCMV arguments carry more than one key point, a coverage-sensitive metric like soft-Recall is the more informative headline number; single-point generators can look artificially good under precision-only reporting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ArgCMV, a key point (KP) extraction benchmark built from ~12K arguments across ~3.1K r/ChangeMyView threads. The authors argue that the established ArgKP21 dataset is unrepresentative of real online debates, and they support this with complexity comparisons (token length, ADU diversity, topic count). Ground-truth KPs are produced by a two-step LLM pipeline (GPT-4o-mini extraction, GPT-4o mapping) with limited human validation. The paper then benchmarks existing KP extraction models and several small language models, reporting that ArgKP21-trained methods degrade on ArgCMV and that open-source SLMs achieve the best few-shot results.

Significance. If the benchmark is valid, ArgCMV is a useful and timely resource: it targets long-context, multi-turn online argumentation, which is absent from existing KP datasets. The authors release code and data, provide explicit train/dev/test splits, and report statistically tested complexity differences. The benchmarking effort is also reasonable in scope, covering both a graph-partitioning baseline and three modern open-weight SLMs. However, the central claim that ArgCMV is harder and that existing methods 'do not adapt well' is measured entirely against LLM-generated gold KPs that are only lightly human-validated. Because the gold labels are the yardstick for every score in Table 4, the benchmark's validity rests on an assumption the paper has not yet adequately established.

major comments (4)
  1. [§5, Step 3; §9 Limitations] The ground-truth KPs are LLM-generated and validated on only 300/12,262 arguments (2.5%). The separate coverage check on ~130 arguments found that 15.7% and 17.1% of arguments have at least one human-identified KP missing from the LLM extraction, with Cohen's κ = 0.33 (minimal agreement). Because soft-recall (Eq. sR) is computed against these possibly incomplete gold KPs, the reported sR values for all models in Table 4 may be systematically suppressed. The paper's own Limitations section acknowledges 'we only validated a limited set which leaves room for some imperfections.' This is load-bearing for the claim that existing methods 'do not adapt well.' Please report the distribution of missing KPs, provide a larger validation sample with corrected gold labels, or otherwise quantify the effect of label incompleteness on the benchmark scores.
  2. [§6.4, Table 4] The authors state: 'we include the results for the run which results in the maximum sF1 value.' Selecting the best of multiple runs overstates performance, especially because temperature is 0.1 rather than 0. No variance or number of runs is reported. This makes the SLM rankings and the comparison to baselines less reliable. Report mean and standard deviation across runs, or all runs; do not select the maximum. The claim that 'we don’t observe large variations across runs' needs supporting evidence.
  3. [§3 and §5, Data collection] All messages authored by the OP are labeled pro (+1) and all other-user messages con (−1). In r/ChangeMyView, however, the OP can concede, change their view, or express agreement, and other users can support the OP's view. The premise that the two author-based pools form 'coherent... mutually opposed' argument groups is therefore not guaranteed. This could introduce stance noise and affect which KPs are extracted and mapped. Please analyze or filter for concession/agreement behavior, or justify the role-based labeling more strongly.
  4. [§6.1, Graph Partitioning baseline] When the adapted graph partitioning algorithm fails to produce any KPs for a topic-stance combination, those topics are excluded from the metrics. This selection can inflate the reported scores for the (Li et al., 2024b) baseline and obscures how often the approach fails on ArgCMV. Report the number/proportion of excluded topics, and consider an evaluation that includes failures (e.g., empty KP sets).
minor comments (5)
  1. [Table 1 / §4] The statistical test used for the token-length and ADU-count comparisons is not stated. Table 1 labels a column 'χ2 pval', but a chi-square test is not the natural test for mean differences. Specify the test for each row (e.g., t-test or Mann-Whitney).
  2. [Figure 2] Typo: 'Rhetotical' should be 'Rhetorical'.
  3. [Table 2] Typo: 'Kirppendorff’s Alpha' should be 'Krippendorff’s Alpha'.
  4. [Appendix A, System Prompt] Typo: 'A key should should not exceed' should be 'A key should not exceed'.
  5. [§5, Step 3] The definitions of KP Precision and KP Recall in the validation are not fully clear. Please give the exact formulas used for these two metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ArgCMV is a newly collected benchmark whose difficulty claims rest on independent statistical comparisons and external baselines, not on a self-referential derivation.

full rationale

The paper's central claims are (i) ArgCMV is more complex than ArgKP21 and (ii) existing methods do not adapt well to ArgCMV. Claim (i) is supported by independent measurements—token length, ADU diversity, topic counts, and chi-square tests—that do not depend on the LLM-generated key points. Claim (ii) is an empirical benchmark result: the authors train/reproduce Li et al. (2024b) models and run open SLMs with fixed prompts, then score them against the released gold key points. The gold key points are produced by GPT-4o-mini extraction and GPT-4o mapping, so benchmark scores are, as with any annotated benchmark, scores against the annotation instrument; but the paper does not fit any parameter to ArgCMV and then 'predict' that same quantity. The admitted limitations—only 300/12,262 arguments human-validated, 15.7–17.1% coverage misses, Cohen's kappa = 0.33—are data-quality/validity concerns, not circularity: incomplete labels affect all models' recall scores but do not by construction force the observed ranking or the ArgKP21-vs-ArgCMV gap. Self-citations (Chandrasekharan group HCI work; Zhan et al. SLM-mod) appear only in motivation/related work and are not load-bearing for the benchmark's derivation. No uniqueness theorem, ansatz-by-citation, or renaming of a known result is present. The derivation chain is self-contained: data collection, LLM annotation, human spot-check, static benchmark construction, and external model evaluation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the validity of LLM-generated key points as ground truth, which is only lightly human-validated. The paper also relies on the ADU tool for complexity comparisons and a simplified stance model. No new physical or conceptual entities are introduced.

free parameters (3)
  • kp_token_length
    The prompt uses a token length limit for generated key points ({kp_token_length} tokens), chosen by the authors; it controls the granularity of the KPs and is not tuned.
  • K-means initial cluster count = half the number of arguments in the graph
    The baseline (Li et al., 2024b) originally used the number of ground truth KPs as the initial cluster count, but since ArgCMV has more KPs than arguments, this was changed to half the argument count. This ad hoc choice affects the baseline's clustering and results.
  • Inference hyperparameters = temperature=0.1, top_p=0.94, max_new_token=256 for SLMs; temperature=0 for GPT models
    Experimental settings chosen for decoding; they affect the outputs of the open-source models and the GPT pipelines.
assumptions (5)
  • domain assumption LLM-generated extracts by gpt-4o-mini and gpt-4o are close enough to human judgments to serve as ground-truth key points.
    Section 5 Steps 1-2 build the dataset using these models; manual validation covers only 300 arguments (2.5%) and the coverage check yields Cohen's Kappa 0.33, so the assumption is not fully verified.
  • domain assumption The ADU annotation model by Mirzakhmedova et al. (2023) correctly labels argumentative discourse units in both ArgKP21 and ArgCMV.
    Section 4 uses this model to compare ADU distributions; errors in the tool would bias the complexity comparison.
  • domain assumption Every message by the original poster is a pro argument (+1) and every message by any other user is a con argument (-1).
    Section 5 Data Collection assigns stance by author role. In r/ChangeMyView, non-OP users can also support the OP's view, and the OP may acknowledge counterarguments, so this simplification may mislabel some arguments.
  • domain assumption The random 80/10/10 topic split prevents topic-level leakage.
    Section 5 Dataset Statistics describes the split but provides no analysis of topic-difficulty balance across splits.
  • domain assumption Rouge and BLEURT-based soft metrics are valid for comparing key point extractors.
    Section 6.2 adopts these metrics from prior work; their adequacy for evaluating KPs with varied wording is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ArgCMV: An Argument Summarization Benchmark for the LLM-era." pith.science (2026). https://pith.science/paper/5EESGUW6

@misc{pith2026250819580,
  author       = {Pith},
  title        = {Pith review of: ArgCMV: An Argument Summarization Benchmark for the LLM-era},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5EESGUW6}},
  note         = {Machine review of arXiv:2508.19580}
}
read the original abstract

Key point extraction is an important task in argument summarization which involves extracting high-level short summaries from arguments. Existing approaches for KP extraction have been mostly evaluated on the popular ArgKP21 dataset. In this paper, we highlight some of the major limitations of the ArgKP21 dataset and demonstrate the need for new benchmarks that are more representative of actual human conversations. Using SoTA large language models (LLMs), we curate a new argument key point extraction dataset called ArgCMV comprising of around 12K arguments from actual online human debates spread across over 3K topics. Our dataset exhibits higher complexity such as longer, co-referencing arguments, higher presence of subjective discourse units, and a larger range of topics over ArgKP21. We show that existing methods do not adapt well to ArgCMV and provide extensive benchmark results by experimenting with existing baselines and latest open source models. This work introduces a novel KP extraction dataset for long-context online discussions, setting the stage for the next generation of LLM-driven summarization research.

Figures

Figures reproduced from arXiv: 2508.19580 by the authors.

Figure 1
Figure 1. An example post on r/ChangeMyView, with green paths highlighting valid dialogues and red paths highlighting invalid branches for our data collection. sively used for summarizing news articles (Zhang et al., 2024a,b), scientific articles (Tang et al., 2023; Van Veen et al., 2023), books (Chang et al., 2023), and dialogues (Tian et al., 2024; Yin et al., 2024; Zhu et al., 2024). We use LLMs for the task for KP extract… view at source ↗
Figure 2
Figure 2. Distribution of ADU units for the two datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Our two-step LLM-based pipeline used to extract key points (KPs) for the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Box-plot showing the distribution of number [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Distribution of topics across 7 high-level cat [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 38 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Milad Alshomary, Timon Gurcke, Shahbaz Syed, Philipp Heinrich, Maximilian Splieth \"o ver, Philipp Cimiano, Martin Potthast, and Henning Wachsmuth. 2021. Key point analysis via contrastive learning and extractive argument summarization. arXiv preprint arXiv:2109.15086

    work page Pith review arXiv 2021

  4. [4]

    Moritz Altemeyer, Steffen Eger, Johannes Daxenberger, Tim Altendorf, Philipp Cimiano, and Benjamin Schiller. 2025. Argument summarization and its evaluation in the era of large language models. arXiv preprint arXiv:2503.00847

    arXiv 2025

  5. [5]

    Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav, and Noam Slonim. 2020 a . From arguments to key points: Towards automatic argument summarization. arXiv preprint arXiv:2005.01619

    arXiv 2020

  6. [6]

    Roy Bar-Haim, Yoav Kantor, Lilach Eden, Roni Friedman, Dan Lahav, and Noam Slonim. 2020 b . Quantitative argument summarization and beyond: Cross-domain key point analysis. arXiv preprint arXiv:2010.05369

    work page Pith review arXiv 2020

  7. [7]

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153

    arXiv 2025

  8. [8]

    Sumit Bhatia, Prakhar Biyani, and Prasenjit Mitra. 2014. Summarizing online forum discussions--can dialog acts of individual messages help? In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2127--2131

    work page 2014

Show all 58 references
  1. [9]

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2023. Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785

    2023 arXiv

  2. [10]

    Liying Cheng, Lidong Bing, Ruidan He, Qian Yu, Yan Zhang, and Luo Si. 2022. Iam: a comprehensive and large-scale dataset for integrated argument mining tasks. arXiv preprint arXiv:2203.12257

    2022 arXiv

  3. [11]

    Frederick Choi, Tanvi Bajpai, Sowmya Pratipati, and Eshwar Chandrasekharan. 2023. Convex: A visual conversation exploration system for discord moderators. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1--30

    2023

  4. [12]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    2024

  5. [13]

    Michael Han Daniel Han and Unsloth team. 2023. http://github.com/unslothai/unsloth Unsloth

    2023

  6. [14]

    Charlie Egan, Advaith Siddharthan, and Adam Wyner. 2016. Summarising the points made in online political debates. In Proceedings of the 3rd Workshop on Argument Mining, The 54th Annual Meeting of the Association for Computational Linguistics, pages 134--143. Association for Co...

    2016

  7. [15]

    Roni Friedman, Lena Dankin, Yufang Hou, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2021. Overview of the 2021 key point analysis shared task. arXiv preprint arXiv:2110.10577

    2021 arXiv

  8. [16]

    Margherita Gambini, Caterina Senette, Tiziano Fagni, and Maurizio Tesconi. 2024. Evaluating large language models for user stance detection on x (twitter). Machine Learning, 113(10):7243--7266

    2024

  9. [17]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7805--7813

    2020

  11. [19]

    Jia Guo, Liying Cheng, Wenxuan Zhang, Stanley Kok, Xin Li, and Lidong Bing. 2023. Aqe: argument quadruplet extraction via a quad-tagging augmented generative approach. arXiv preprint arXiv:2305.19902

    2023 arXiv

  12. [20]

    Christopher Hidey and Kathleen McKeown. 2018. Persuasive influence detection: The role of argument sequencing. In Proceedings of the AAAI conference on artificial intelligence, volume 32

    2018

  13. [21]

    John Hughes. 2021. krippendorffsalpha: An r package for measuring agreement using krippendorff's alpha coefficient. arXiv preprint arXiv:2103.12170

    2021 arXiv

  14. [22]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  15. [23]

    Jane Im, Sonali Tandon, Eshwar Chandrasekharan, Taylor Denby, and Eric Gilbert. 2020. Synthesized social signals: Computationally-derived social signals from account histories. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1--12

    2020

  16. [24]

    Manav Nitin Kapadnis, Sohan Patnaik, Siba Smarak Panigrahi, Varun Madhavan, and Abhilash Nandy. 2021. Team enigma at argmining-emnlp 2021: Leveraging pre-trained language models for key point matching. arXiv preprint arXiv:2110.12370

    2021 arXiv

  17. [25]

    Aman Kumar, Amit Shankar, Aviral Kumar Tiwari, and Hae-Jung Hong. 2023. Understanding dark side of online community engagement: an innovation resistance theory perspective. Information Systems and e-Business Management, pages 1--27

    2023

  18. [26]

    Sung-Chul Lee, Jaeyoon Song, Eun-Young Ko, Seongho Park, Jihee Kim, and Juho Kim. 2020. https://doi.org/10.1145/3313831.3376609 Solutionchat: Real-time moderator support for chat-based structured discussion . In Proceedings of the 2020 CHI Conference on Human Factors in Comput...

    2020

  19. [27]

    Hao Li, Viktor Schlegel, Riza Batista-Navarro, and Goran Nenadic. 2023. Do you hear the people sing? key point analysis via iterative clustering and abstractive summarisation. arXiv preprint arXiv:2305.16000

    2023 arXiv

  20. [28]

    Hao Li, Yuping Wu, Viktor Schlegel, Riza Batista-Navarro, Tharindu Madusanka, Iqra Zahid, Jiayan Zeng, Xiaochi Wang, Xinran He, Yizhi Li, et al. 2024 a . Which side are you on? a multi-task dataset for end-to-end argument summarisation and evaluation. arXiv preprint arXiv:2406.03151

    2024 arXiv

  21. [29]

    Xiao Li, Yong Jiang, Shen Huang, Pengjun Xie, Gong Cheng, and Fei Huang. 2024 b . Exploring key point analysis with pairwise generation and graph partitioning. arXiv preprint arXiv:2404.11384

    2024 arXiv

  22. [30]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  23. [31]

    Yijun Liu, Frederick Choi, and Eshwar Chandrasekharan. 2025. Needling through the threads: A visualization tool for navigating threaded online discussions. arXiv preprint arXiv:2506.11276

    2025

  24. [32]

    Guillermo Marco, Luz Rello, and Julio Gonzalo. 2024. Small language models can outperform humans in short creative writing: A study comparing slms with humans and llms. arXiv preprint arXiv:2409.11547

    2024 arXiv

  25. [33]

    Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276--282

    2012

  26. [34]

    Nailia Mirzakhmedova, Johannes Kiesel, Khalid Al-Khatib, and Benno Stein. 2023. Unveiling the power of argument arrangement in online persuasive discussions. In 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 15659--15671. Association fo...

    2023

  27. [35]

    MistralAI and NVIDIA. 2024. Mistral-nemo-instruct-2407. https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407. Accessed: 2025-05-20

    2024

  28. [36]

    Gaku Morio, Ryo Egawa, and Katsuhide Fujita. 2019. Revealing and predicting online persuasion strategy with elementary units. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag...

    2019

  29. [37]

    Karl Pearson. 1900. X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine ...

    1900

  30. [38]

    Allen Roush and Arvind Balaji. 2020. Debatesum: A large-scale argument mining and summarization dataset. arXiv preprint arXiv:2011.07251

    2020 arXiv

  31. [39]

    Allen Roush, Yusuf Shabazz, Arvind Balaji, Peter Zhang, Stefano Mezza, Markus Zhang, Sanjay Basu, Sriram Vishwanath, and Ravid Shwartz-Ziv. 2024. Opendebateevidence: A massive-scale argument mining and summarization dataset. Advances in Neural Information Processing Systems, 3...

    2024

  32. [40]

    Timo Schick and Hinrich Sch \"u tze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint arXiv:2009.07118

    2020 arXiv

  33. [41]

    Charlotte Schluger, Jonathan P Chang, Cristian Danescu-Niculescu-Mizil, and Karen Levy. 2022. Proactive moderation of online discussions: Existing practices and the potential for algorithmic support. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):1--27

    2022

  34. [42]

    Patrick Schober, Christa Boer, and Lothar A Schwarte. 2018. Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia, 126(5):1763--1768

    2018

  35. [43]

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696

    2020 arXiv

  36. [44]

    Chenhao Tan, Vlad Niculae, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Winning arguments: Interaction dynamics and persuasion strategies in good-faith online discussions. In Proceedings of the 25th international conference on world wide web, pages 613--624

    2016

  37. [45]

    Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. 2023. Evaluating large language models on medical evidence summarization. NPJ digital medicine, 6(1):158

    2023

  38. [46]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  39. [47]

    Yuanhe Tian, Fei Xia, and Yan Song. 2024. Dialogue summarization with mixture of experts based on large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7143--7155

    2024

  40. [48]

    Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, et al. 2023. Clinical text summarization: adapting large language models can outperform human expert...

    2023

  41. [49]

    Anar Yeginbergen, Maite Oronoz, and Rodrigo Agerri. 2025. Dynamic knowledge integration for evidence-driven counter-argument generation with large language models. arXiv preprint arXiv:2503.05328

    2025 arXiv

  42. [50]

    Yuan-Jhe Yin, Bo-Yu Chen, and Berlin Chen. 2024. A novel llm-based two-stage summarization approach for long dialogues. In 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1--6. IEEE

    2024

  43. [51]

    Xianyang Zhan, Agam Goyal, Yilun Chen, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Slm-mod: Small language models surpass llms at content moderation. arXiv preprint arXiv:2410.13155

    2024 arXiv

  44. [52]

    Amy X Zhang and Justin Cranshaw. 2018. Making sense of group chat through collaborative tagging and summarization. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1--27

    2018

  45. [53]

    Amy X Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging discussion forums and wikis using recursive summarization. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, pages 2082--2096

    2017

  46. [54]

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024 a . Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39--57

    2024

  47. [55]

    Yang Zhang, Hanlei Jin, Dan Meng, Jun Wang, and Jinghua Tan. 2024 b . A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901

    2024

  48. [56]

    Chenye Zhao, Yingjie Li, Cornelia Caragea, and Yue Zhang. 2024. Zerostance: Leveraging chatgpt for open-domain stance detection via dataset generation. In Findings of the Association for Computational Linguistics ACL 2024, pages 13390--13405

    2024

  49. [57]

    Rongxin Zhu, Jey Han Lau, and Jianzhong Qi. 2024. Factual dialogue summarization via learning from large language models. arXiv preprint arXiv:2406.14709

    2024 arXiv

  50. [58]

    Timon Ziegenbein, Shahbaz Syed, Martin Potthast, and Henning Wachsmuth. 2024. Objective argument summarization in search. In Conference on Advances in Robust Argumentation Machines, pages 335--351. Springer Nature Switzerland Cham

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.