LLMs for Argument Mining: Detection, Extraction, and Relationship Classification of pre-defined Arguments in Online Comments

Eduard Hovy; Lea Frermann; Matteo Guida; Yulia Otmakhova

REVIEW 4 major objections 4 minor 13 references

LLMs for Argument Mining: Detection, Extraction, and Relationship Classification of pre-defined Arguments in Online Comments

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can mine online arguments, but emotion trips them up

desk verdict Useful first systematic LLM evaluation for pre-defined argument mining, but the task-to-dataset mapping in Section 3.2 is self-contradictory and must be fixed before the numbers can be trusted. read the letter →

arxiv 2505.22956 v1 pith:TVFJVQEF submitted 2025-05-29 cs.CL

Matteo Guida , Yulia Otmakhova , Eduard Hovy , Lea Frermann This is my paper

classification cs.CL

keywords argumentmininglargelanguagemodelsonlinecommentsdetectionspanextractionrelationshipclassificationpre-definedargumentserroranalysis

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can reliably mine pre-defined arguments from online comments on contested topics. It splits the job into three tasks—detecting whether a comment uses a given argument, extracting the exact span that expresses it, and classifying whether the comment supports or attacks it—and evaluates four LLMs on over 2,000 comments across six polarizing issues. The headline result is that large and fine-tuned LLMs do well, often outperforming a fine-tuned RoBERTa baseline, especially at binary support/attack classification. But the error analysis shows the models systematically over-predict arguments in emotionally charged or sarcastic comments and degrade on longer, more nuanced comments, which would distort any moderation or opinion-analysis system built on them.

What carries the argument

The central machinery is the combination of (1) pre-defined argument lists drawn from debate platforms (following Levy et al., 2014), (2) three atomic tasks that separate detection, extraction, and relationship classification, and (3) a systematic comparison of zero-, one-, and five-shot prompting against fine-tuned baselines, with prompt variants tuned per task. The error analysis then becomes the interpretative tool: by splitting errors into false positives versus false negatives and correlating them with comment length and emotional content, the paper localizes where the models fail. The fine-tuning of Llama3 with LoRA, and the use of RoBERTa as a strong conventional baseline, are what let the paper attribute gains to model scale and adaptation rather than to prompt engineering alone.

What would settle it

Re-annotate a stratified random sample of comments—especially the long and emotionally charged ones—with expert annotators using the original argument lists, and compare the resulting labels to both the original gold and the model predictions. If the original gold often missed arguments that experts find, or if models agree with experts more than with the original gold, then the reported over-prediction and length effects are partly artifacts of noisy labels rather than model biases.

Watch

Extended reading notes

Core claim

The paper's central claim is that the task decomposition works and reveals a split: LLMs are strong at classifying whether a comment supports or attacks a known argument (near-perfect for the largest models on the binary version), moderately strong at detecting whether a pre-defined argument is used at all, and weaker at extracting the exact text span when measured by ROUGE-L, though semantic evaluation with BERTScore and manual inspection suggests the true extraction ability is higher than the lexical metric indicates. The same decomposition exposes systematic weaknesses: about 66% of detection errors are false positives, concentrated on arguments that oppose the topic and on emotionally charged comments, and accuracy falls off significantly with comment length. On the paper's own terms, the conclusion is that LLMs show promise for large-scale argument mining with pre-defined arguments, but their bias toward strong language and away from extended reasoning makes them unsafe for direct deployment in content moderation or public opinion analysis without further work.

Load-bearing premise

The evaluation rests on the gold labels in the COMARG and YRU datasets being accurate and consistent enough to score model performance; COMARG has only moderate inter-annotator agreement with majority-vote labels, and YRU sentence-level annotations are projected to comment level by treating any unlabeled sentence as absence.

Editorial extensions

If this is right

Fine-tuned LLMs such as Llama3 with LoRA give the best detection and extraction performance among the tested models, beating both prompted LLMs and RoBERTa, but at a substantial environmental cost.
Large prompted models like GPT-4o and Gemini are nearly perfect at binary support/attack classification, so this part of the pipeline could already be automated reliably on short, clear comments.
Systems that mine or moderate opinions should not be deployed without handling of long and emotional comments, because false positives concentrate exactly there, especially for arguments that oppose a topic.
Few-shot examples do not reliably improve classification performance; they help only for span extraction, so prompt design should be per-task rather than assumed universally beneficial.
The reported ROUGE-L scores understate extraction ability; evaluating with semantic metrics such as BERTScore and inspecting outputs yields a more favorable picture, meaning extraction results should not be read as hard ceilings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The systematic false-positive bias toward Con arguments and emotional language could arise from the models' prior on what argumentative text looks like; a direct test would be to run the same tasks on de-biased prompts or on comments stripped of emotion-indicating words and see whether the error distribution shifts.
The length effect suggests a possible attention or recency bottleneck: the models may be keying on only part of a comment; chunking comments into clauses and aggregating predictions might recover missed arguments, an extension the paper does not test.
Because the datasets date from 2014, the argument inventories may be stale; re-running the same protocol on current debate content would show whether the reported strengths and weaknesses are stable or an artifact of those older topics.
The near-ceiling binary relationship classification may reflect that support/attack is easier than detection itself; this inverts the common assumption that detection must precede relation classification, hinting that joint or end-to-end models should be evaluated differently.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the useful part: this is the first systematic test of four LLMs on all three pre-defined-argument tasks (detection, span extraction, support/attack classification) across COMARG and YRU. The error analysis is a real plus—finding that false positives dominate, that length hurts, and that emotional language triggers over-prediction is exactly the kind of practical information the field needs. The BERTScore check for extraction, though only for the best model, is a sensible sanity check on Rouge-L.

Now the soft spot that matters most. Section 3.2 and Section 3.3/Table 2/Table 3 disagree about which dataset feeds which task. 3.2 says Task 2 runs on "the four COMARG topics" (COMARG has two topics) and Task 3 runs on "the two topics in this dataset" (YRU has four). 3.3 and the table headers say the opposite: Task 2 uses YRU (AB, GR, MA, OB), Task 3 uses COMARG (GM, UG). The tables are internally consistent, so the numbers are probably fine, but the text is not. A reader cannot tell from the paper alone which comments were scored for extraction or relationship classification without reverse-engineering the column headers. That is a provenance problem, not a stylistic one, and it needs fixing in revision. It is also more immediate than the annotation-noise concern—no amount of gold-label noise explains a contradiction about what data was used.

The reader's worry about gold-label noise has some force: COMARG's inter-annotator agreement is only moderate, and YRU's sentence-to-comment projection could introduce errors. But that is a standard limitation for evaluation papers on legacy data, and the paper acknowledges it. I would not treat it as a fatal flaw.

Other soft spots are minor: no significance tests between models, just standard deviations; the anonymous repo lacks a commit hash, so reproducibility is not yet verified; only Llama3 gets fine-tuned. None of these undercut the main conclusions.

The paper is for argument-mining researchers and anyone building opinion-analysis or moderation systems on top of LLMs. It deserves a serious referee. I'd send it out with a request for major revision—rewrite the task/data mapping in 3.2, add significance tests where feasible, and pin the code. As is, I wouldn't trust the text, but the underlying measurements look worth publishing.

Referee Report

4 major / 4 minor

Summary. The paper evaluates four LLMs (Llama3-8b-Instruct, GPT4o, GPT4o-mini, Gemini1.5-Flash) on three argument mining tasks over the COMARG and YRU datasets: binary argument detection (Task 1), span extraction (Task 2), and support/attack relationship classification, including an explicit/implicit four-way variant (Task 3). The authors compare prompted LLMs, a fine-tuned RoBERTa baseline, and a LoRA fine-tuned Llama3, and they report a detailed error analysis of the strongest prompt-based model. Their main findings are that fine-tuned Llama3 outperforms all other models on Tasks 1 and 2, prompted LLMs are competitive on classification tasks, and error analysis reveals systematic over-prediction on emotional language and degraded performance on long comments. The paper also releases code, data, and prompts.

Significance. If the results hold, this is a useful empirical contribution to the growing literature on LLM evaluation for argument mining. The paper is valuable for separating detection, extraction, and relation classification into distinct tasks, and for combining both traditional baselines and modern LLMs in one study. The released code and prompts support reproducibility. The error analysis highlights actionable failure modes (emotional language, long comments) that are relevant to downstream applications such as content moderation and opinion analysis. However, the current manuscript contains serious internal inconsistencies about which dataset feeds which task, which must be resolved before the results can be interpreted or verified.

major comments (4)

[§3.2, §3.3, Tables 2 and 3] The task-to-dataset mapping is internally contradictory and prevents reproduction of the main results. Section 3.2 states that Task 2 (span extraction) is evaluated on COMARG and that COMARG has manually annotated argument spans; it then says "the four COMARG topics," although COMARG has only two topics (GM, UGIP) and, per Section 3.1, it is YRU that is labeled with argument spans. Section 3.3 states the opposite for Task 2, namely that only YRU labels were considered. Table 2's columns (AB, GR, MA, OB) are YRU topics, which matches Section 3.3 but not Section 3.2. The identical reversal occurs for Task 3: Section 3.2 assigns relationship classification to YRU, but Table 3 reports results on GM and UG, which are COMARG topics, consistent with Section 3.3. These contradictions mean the provenance of the numbers in Tables 2 and 3 cannot be determined from the text as written, so the corresponding claims about extraction and relation-classification performance are not currently verifiable.
[§4.1, §4.2, Tables 1 and 2] The central comparative claim that fine-tuned Llama3 "outperformed all other models" on Tasks 1 and 2 rests on small score differences (e.g., 0.76 vs. 0.73 macro-F1 on the combined Task 1 set; 0.54 vs. 0.51 ROUGE-L on Task 2). The paper reports no significance tests, confidence intervals, or paired comparisons for these differences. The reported standard deviations for few-shot runs capture variation across example sets but not across test folds or model seeds, and no variation is reported for zero-shot results. Without such tests, a difference of 0.01–0.03 cannot support a strong superiority claim. The paper should either add significance testing or soften the comparison to a descriptive observation.
[§4.4, Abstract, Conclusion] The error analysis is conducted exclusively on a single model (Gemini 1.5-Flash, 5-shot), but the abstract and conclusion generalize the findings to LLMs as a class, e.g., "LLMs frequently over-predict arguments" and "current limitations of LLMs." No evidence is provided that GPT-4o, GPT-4o-mini, or fine-tuned Llama3 exhibit the same false-positive patterns, length effects, or emotional-language biases. The generalized claims about systematic shortcomings should be scoped to the specific model analyzed, or the error analysis should be extended to at least one additional model family.
[Abstract, §5] The abstract states that the approach comes "at a significant environmental cost," and the conclusion repeats this, but the manuscript contains no measurement or citation to quantify energy consumption or carbon emissions for any of the reported experiments. Since this is presented as one of the main trade-offs of the approach, the manuscript should either provide a quantitative estimate (e.g., kWh or CO2e per experiment) or clearly frame the cost as a qualitative concern rather than a measured result.

minor comments (4)

[Appendices F–I] Appendix Sections F, G, H, and I duplicate, nearly verbatim, the content of Sections B, C, D, and E. The duplicated sections should be removed or consolidated.
[Table 12] In the prompt for Task 3 (full scale), the classification rules state "Use ONLY specified labels (1 or 5)" but then list labels 1, 2, 4, and 5, and the output schema says "must be 1, 2, 4 or 5". The instruction is internally inconsistent and should be corrected to reference all four labels.
[§4.4, Figure 4] The t-test for Task 1 length differences is reported as (t = −12.103, p << 0.001), but the degrees of freedom and whether the test is paired or independent are not stated. Please provide full test statistics and a precise p-value.
[§4.2] BERTScore validation is computed only for Gemini, which is described as the "best-performing model." However, the paper's overall best model on Task 2 is fine-tuned Llama3, so the semantic-validity check does not cover the model that the main claim highlights. The BERTScore analysis should either be extended to fine-tuned Llama3 or explicitly scoped as a check on the best prompt-based model only.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation against external gold labels, with held-out cross-validation and no fitted parameter masquerading as a prediction.

full rationale

This paper reports an empirical evaluation of four LLMs on three argument-mining tasks using two externally annotated corpora (COMARG from Boltuži?c and Šnajder 2014 and YRU from Hasan and Ng 2014). The claimed results are measured model outputs against gold labels, not quantities derived from the model's own fitted parameters. Fine-tuned Llama3 and RoBERTa are trained with five-fold stratified cross-validation, so test predictions come from held-out folds rather than from the training labels being re-predicted. No parameter is fitted to a subset of the data and then renamed as a prediction for a closely related quantity. The only dataset-related citations are attributions to the original 2014 corpus creators, not self-citations by the current authors, and no load-bearing argument reduces to a self-citation chain. There is no invoked uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known empirical pattern as a new derivation. The internal inconsistency noted in the skeptic review, where Section 3.2 says Task 2 is evaluated on COMARG topics while Section 3.3 and Table 2 refer to YRU topics, is a reproducibility and correctness concern about which corpus was actually scored, but it is not a circularity: the model outputs are still compared against external gold spans, whichever corpus those spans come from. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the quality of existing gold annotations, the representativeness of the pre-defined argument lists, and the stability of LLM decoding choices. No new entities are introduced. The hyperparameters listed are standard choices, not fitted to the target results.

free parameters (4)

LoRA rank (r) = 16
Chosen in Appendix E for parameter-efficient fine-tuning of Llama3; this controls the capacity of the adapter and directly affects the fine-tuned model's performance on Task 1 and Task 2.
LoRA alpha = 16
Scaling factor for LoRA updates in Appendix E; hand-set and not tuned per task.
Learning rate = 2e-4
Used for LoRA fine-tuning in Appendix E; a standard choice that affects the final model.
Fine-tuning epochs = 3 for Task 1, 5 for Task 2
Set in Appendix E with training stopped when loss approached zero; affects the fine-tuned Llama3 results.

assumptions (4)

domain assumption Gold labels in COMARG and YRU are accurate and consistent enough to score LLM performance.
Section 3.1 reports moderate inter-annotator agreement for COMARG and moderate to high for YRU, with majority vote or discussion resolving disagreements. All quantitative scores depend on these labels.
domain assumption Projecting YRU sentence-level argument labels to comment level preserves the presence signal for Task 1.
Section 3.3 states that arguments not identified in any sentence are labeled absent, which may over-count absences if a span was missed.
domain assumption Setting temperature to 0 and top_p to 1 makes LLM outputs stable enough for evaluation.
Section 3.4 follows prior practice to minimize variability, but proprietary APIs can still introduce non-determinism.
domain assumption The 2014-era pre-defined arguments remain a valid test of current argument use despite social shifts.
Section 6 acknowledges gay marriage arguments may be outdated after US legalization in 2015, which could affect relevance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLMs for Argument Mining: Detection, Extraction, and Relationship Classification of pre-defined Arguments in Online Comments." pith.science (2026). https://pith.science/paper/TVFJVQEF

@misc{pith2026250522956,
  author       = {Pith},
  title        = {Pith review of: LLMs for Argument Mining: Detection, Extraction, and Relationship Classification of pre-defined Arguments in Online Comments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TVFJVQEF}},
  note         = {Machine review of arXiv:2505.22956}
}

read the original abstract

Automated large-scale analysis of public discussions around contested issues like abortion requires detecting and understanding the use of arguments. While Large Language Models (LLMs) have shown promise in language processing tasks, their performance in mining topic-specific, pre-defined arguments in online comments remains underexplored. We evaluate four state-of-the-art LLMs on three argument mining tasks using datasets comprising over 2,000 opinion comments across six polarizing topics. Quantitative evaluation suggests an overall strong performance across the three tasks, especially for large and fine-tuned LLMs, albeit at a significant environmental cost. However, a detailed error analysis revealed systematic shortcomings on long and nuanced comments and emotionally charged language, raising concerns for downstream applications like content moderation or opinion analysis. Our results highlight both the promise and current limitations of LLMs for automated argument analysis in online comments.

Figures

Figures reproduced from arXiv: 2505.22956 by the authors.

**Figure 2.** A comment (top, left) and pre-defined argu [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 4.** The effect of comment length on comment identification accuracy (Task 1; Violin/box plots) and argument extraction (Task 2; Rouge-L). accounting for approximately 66% of all errors. This pattern is particularly strong for Con arguments (which are against a topic), where 76% of all errors are FPs (62% for Pro arguments, in support of a topic). In other words: argumentative content is systematically over-predicted in… view at source ↗

Figures from the paper (1 more)

**Figure 3.** Figure 3: Proportion of false positive and false negative [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

[1]

Determine if the comment explicitly or implicitly uses the given argument

work page
[2]

id": "id

Assign a binary label: - 1 if the argument is present - 0 if the argument is not present Requirements: - Only use 1 or 0 as labels - Provide output in valid JSON format - Do not repeat or include the input text in the response - Focus solely on the presence/absence of the specific argument Return your analysis in this exact JSON format: "id": "id", "label...

work page
[3]

Extract the precise text span

work page
[4]

id": "{id}

Format the output according to specifications Critical Requirements: - Extract EXACT text only (no paraphrasing) - Include COMPLETE relevant phrases - Use MINIMUM necessary context - Maintain ORIGINAL formatting - Return V ALID JSON only Output Schema: { "id": "{id}", "span": "exact_text_from_comment" # must be verbatim quote } Input Text: Table 10: Promp...

work page
[5]

Read the input text carefully

work page
[6]

Locate exact text spans that: - Directly reference the target argument - Express the same idea as the argument

work page
[12]

id": "{id}

Format the output according to specifications Classification Rules: - Label = 5: Comment supports/agrees with argument - Label = 1: Comment attacks/disagrees with argument Critical Requirements: - Use ONLY specified labels (1 or 5) - Do NOT quote or repeat input texts - Return V ALID JSON only Output Schema: { "id": "{id}", "label": label_value # must be ...

work page
[13]

Read the input text thoroughly

work page

Show all 13 references

[14]

Evaluate the text’s relationship to the target argument, examining: - Direct support or opposition - Implicit agreement or disagreement
[15]

Make a binary classification decision
[16]

id": "{id}

Format the output according to specifications Classification Rules: - Label = 5: Comment supports/agrees with argument - Label = 4: Comment supports/agrees with argument implicitly/indirectly - Label = 2: Comment attacks/disagrees with argument implicitly/indirectly - Label = ...
[2023]

Under God

Investigating the impact of decoding strategies on large language model performance: A systematic analysis. arXiv preprint arXiv:2306.09265. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Pre...

2020 arXiv
[2024]

ArXiv:2402.11243 [cs]

Can Large Language Models perform Relation-based Argument Mining? arXiv preprint. ArXiv:2402.11243 [cs]. Ivan Habernal and Iryna Gurevych. 2017. Argumenta- tion Mining in User-Generated Web Discourse. Com- putational Linguistics, 43(1):125–179. Kazi Saidul Hasan and Vincent Ng...

2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Determine if the comment explicitly or implicitly uses the given argument

work page

[2] [2]

id": "id

Assign a binary label: - 1 if the argument is present - 0 if the argument is not present Requirements: - Only use 1 or 0 as labels - Provide output in valid JSON format - Do not repeat or include the input text in the response - Focus solely on the presence/absence of the specific argument Return your analysis in this exact JSON format: "id": "id", "label...

work page

[3] [3]

Extract the precise text span

work page

[4] [4]

id": "{id}

Format the output according to specifications Critical Requirements: - Extract EXACT text only (no paraphrasing) - Include COMPLETE relevant phrases - Use MINIMUM necessary context - Maintain ORIGINAL formatting - Return V ALID JSON only Output Schema: { "id": "{id}", "span": "exact_text_from_comment" # must be verbatim quote } Input Text: Table 10: Promp...

work page

[5] [5]

Read the input text carefully

work page

[6] [6]

Locate exact text spans that: - Directly reference the target argument - Express the same idea as the argument

work page

[7] [12]

id": "{id}

Format the output according to specifications Classification Rules: - Label = 5: Comment supports/agrees with argument - Label = 1: Comment attacks/disagrees with argument Critical Requirements: - Use ONLY specified labels (1 or 5) - Do NOT quote or repeat input texts - Return V ALID JSON only Output Schema: { "id": "{id}", "label": label_value # must be ...

work page

[8] [13]

Read the input text thoroughly

work page