SLURG: Investigating the Feasibility of Generating Synthetic Online Fallacious Discourse

Cal Blanco; Chelsey Rush; Gavin Dsouza; Hugo Lin

REVIEW 4 major objections 4 minor 14 references

SLURG: Investigating the Feasibility of Generating Synthetic Online Fallacious Discourse

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that LLMs can replicate the syntactic patterns of real online forum comments and that few-shot prompting with gold-labeled examples improves the vocabulary diversity of the generated synthetic comments.

desk verdict A modest, honest feasibility study on generating fallacious forum comments whose vocabulary-diversity finding is believable, but the annotation reliability and missing statistics keep it short of being citable as-is. read the letter →

arxiv 2504.12466 v1 pith:RT2GJPR5 submitted 2025-04-16 cs.CL

Cal Blanco , Gavin Dsouza , Hugo Lin , Chelsey Rush This is my paper · ORCID

classification cs.CL

keywords syntheticdatafallacydetectionlargelanguagemodelsfew-shotpromptingReddit4chanhapaxlegomenavocabularydiversity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can generate synthetic online forum comments that are close enough to real Reddit and 4chan posts to serve as training data for fallacy detection. The authors scrape comments about the Russia-Ukraine conflict, annotate them for three top-level fallacy types using a unified taxonomy, and then prompt DeepHermes-3-Mistral-24B to write new comments with and without annotated few-shot examples. They report that the model reproduces the syntactic patterns of the real data across all prompting conditions, and that including more gold-labeled examples raises the hapax-legomena ratio of the generated text toward the level of genuine forum posts. The result matters because existing fallacy-detection datasets come from formal domains such as political debates and news, and synthetic data could fill the gap for informal online discourse.

What carries the argument

The load-bearing machinery is a three-part setup: (1) the unified hierarchical taxonomy of fallacies from Helwe et al. (2023), used to annotate spans as credibility, logic, or emotion fallacies; (2) XML-tagged few-shot prompts that give the model annotated examples plus explicit fallacy definitions; and (3) the hapax-legomena ratio, the share of tokens appearing once per sentence, as a quantitative proxy for authorial vocabulary diversity. The taxonomy gives the annotation task a boundary, the prompts carry the gold-labeled style information into generation, and the ratio measures how close synthetic text comes to real user style.

What would settle it

A blind test in which human readers are shown pairs of real and generated comments and asked which is human-written; if the readers identify the generated comments at a rate significantly above chance, the claim of syntactic replication is falsified.

Watch

Extended reading notes

Core claim

The central claim is that LLM-generated synthetic fallacious comments are feasible for informal online domains: DeepHermes-3-Mistral-24B reproduces the distribution of noun, verb, prepositional, and subclause phrases found in real Reddit and 4chan comments, and few-shot prompting with gold-labeled examples improves the vocabulary diversity of generated text, measured by hapax-legomena ratio, from a mean of 0.762 to 0.852 and closer clustering near 100%. The paper also shows that annotation few-shot prompting with 20% of gold labels yields the best annotation F1 and the highest qualitative scores for realism and fallacy accuracy, while larger few-shot sets improve span accuracy.

Load-bearing premise

The human annotations used as gold labels and few-shot prompts are accurate enough to serve as ground truth, despite one annotator's low agreement with the others and the small size of the filtered gold set.

Editorial extensions

If this is right

If the finding holds, synthetic comment generation could cheaply expand fallacy-detection datasets for informal domains beyond what human annotation alone can support.
Few-shot prompting strategies that include annotated gold labels should be preferred over zero-shot generation when the goal is vocabulary diversity close to real users.
Benchmarks built from real forum data should account for the possibility that some 'real' posts are themselves LLM-generated, since the paper notices such suspicious posts in its scraped data.
The annotation-F1 results suggest that selection of few-shot split size matters: 20% outperformed both zero-shot and 30% in this setup, so prompt-set size is a tunable factor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next test, not run here, would be to train a fallacy classifier on the synthetic data and measure its transfer performance on held-out real comments; positive transfer would make the feasibility claim actionable for downstream detection.
The hapax-legomena result might be partially driven by the model avoiding repetition rather than matching human lexis; a paired comparison controlling for topic and length could separate these effects.
If synthetic content is already circulating in the scraped forums, the 'real' baseline may itself be contaminated, which would change the interpretation of the diversity gap the paper measures.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the headline result—that few-shot prompting with gold-labeled examples moves the hapax-legomena ratio of generated 4chan/Reddit-style comments from 0.76 to 0.85, toward the 0.87–0.89 of real data—is a real empirical observation worth reporting. Second, the annotation chain underneath the 'fallacious' part of the claim is the weakest link, and the stress-test note mostly lands: the same four annotators who produced the noisy gold labels (one with pairwise Jaccard IAA around 0.52) are the ones who score the generated data for realism, fallacy accuracy, and span accuracy, with no stability analysis. That said, the vocabulary finding does not collapse, because it is about style transfer and would survive even if some imperfectly labeled spans are used as few-shot examples.

What is genuinely new: applying LLM-based synthetic generation to informal, profane, fallacy-laden forum discourse about the Ukraine conflict, and using hapax legomena as a quantitative vocabulary-diversity target rather than relying on qualitative impressions. The paper is honest about its limitations, and the ethical section is thoughtful. The syntactic-similarity claim, however, is based on visual inspection of phrase-type distributions and needs a real distance metric or at least error bars.

Soft spots, in proportion. (1) Gold labels: only 150 samples per annotator, IAA is moderate, the IAA>0.80 filter leaves a small set, and randomly selecting one annotator's labels injects avoidable noise. Fresh annotators or a consensus pass would fix most of this. (2) Statistics: hapax ratios are reported as means without variance or significance tests; we do not know if the 0.76-to-0.85 move is robust across generation seeds. (3) Same-annotator qualitative evaluation confounds the human scores. (4) No data or code released, limiting reproducibility, and the F1 numbers are low enough that 'annotation performance' is a stretch. These are addressable, not load-bearing flaws in the central feasibility direction.

This is a modest feasibility study, not a breakthrough. It deserves a serious referee—workshop or short-paper level—but only after error bars, artifacts, and either fresh annotators or a stricter scoping of claims. I'd bring it to a reading group as an example of an honest, cheap feasibility study; I wouldn't cite it yet.

Referee Report

4 major / 4 minor

Summary. The paper introduces SLURG, a pipeline for generating synthetic fallacious forum-style comments about the Ukraine-Russia conflict using DeepHermes-3-Mistral-24B. The authors scrape Reddit and 4chan data, annotate 150 samples per annotator with three top-level fallacy categories from the Helwe et al. (2023) taxonomy, construct gold splits with varying few-shot proportions, and evaluate both LLM-based fallacy annotation and synthetic generation. The reported results indicate that few-shot prompting, particularly with 20% of the gold-labeled annotations, improves relaxed F1 and vocabulary diversity as measured by the hapax-legomena ratio, and that the generated data roughly matches the phrase-type distribution of real forum comments. The central claim is that generating synthetic fallacious online discourse is feasible and that high-quality few-shot prompts improve stylistic mimicry.

Significance. If the central finding holds, the paper makes a modest but useful contribution to synthetic data generation for fallacy detection in informal, social-media-like domains, where existing datasets are largely formal. The use of external real data for vocabulary and syntax comparisons is a sounder evaluation strategy than self-reported realism alone, and the paper is honest about its limitations, including annotator subjectivity and domain narrowness. The hapax-legomena results are suggestive and align with the claimed feasibility. However, the significance is limited by the use of a single model, a small annotation effort, and the absence of statistical validation; the current evidence is not yet strong enough to establish the feasibility claim beyond reasonable doubt.

major comments (4)

[§3.2–3.3, Figure 5, Table 2] The gold-label construction is not sufficiently reliable to support the F1 evaluation. One annotator has pairwise Jaccard IAA of only 0.51–0.53, and the gold set is created by filtering to samples with IAA greater than 0.80 and then randomly selecting one annotator's annotation. The paper does not report how many samples survive this filter, which is critical because only 150 samples were annotated per annotator. If the surviving gold set is very small, the F1 scores in Table 2 are computed on an unstable reference, and the few-shot examples may themselves teach the model incorrect spans. Please report the gold-set size, the distribution of IAA scores over samples, and preferably replace random selection with adjudicated or consensus labels.
[§5.2, Figure 8] The qualitative evaluation of generated data is performed by the same annotators whose disagreement was documented in §3.2. Since the annotators knew the annotation guidelines and had already formed opinions about the source data, their ratings of realism, fallacy accuracy, and span accuracy are not an independent validation of the generated data. The claim that the 80/20 split receives the highest qualitative scores would be much stronger if evaluated by fresh annotators who are blind to the generation condition, or if inter-annotator agreement on the qualitative task itself were reported. Without this, the qualitative results are confounded by the annotators' prior exposure and subjectivity, as the Limitations section itself concedes.
[§5.1, Table 2] The annotation-performance comparison lacks any measure of uncertainty. The differences across splits are small—for example, strict F1 ranges from 0.2209 to 0.2900 and relaxed F1 from 0.2977 to 0.3897—and no confidence intervals, bootstrap resamples, or significance tests are provided. As a result, the statements that few-shot prompting 'resulted in increased' scores only for 20% and 'declined' for 30% are not supported. Please add error bars or statistical tests (for example, McNemar's test or bootstrap over samples) to determine whether the observed differences are meaningful.
[§5.3, Figures 9–10] The vocabulary-diversity and syntactic-similarity claims are based on visual inspection and aggregate means without error bars or statistical comparisons. The hapax-legomena means for real Reddit and 4chan data are 0.887 and 0.872, while the best few-shot condition reaches 0.852; the paper does not test whether this difference is significant, nor does it report the variance or sample sizes behind these means. Similarly, the claim that 'each of the datasets exhibited very similar distributions' of phrase types rests on eyeballing Figure 10. Please provide quantitative divergence measures or permutation tests, and report the distribution of hapax-legomena values rather than only means.

minor comments (4)

[Throughout] The manuscript contains several typographical errors, including 'real-word' for 'real-world', 'Noteably' for 'Notably', 'the the' in §5.2, and 'generizability' in §5.4. These should be corrected in a revision.
[Figure 8 and surrounding text] The text in §5.2 refers to 'Figure 8' before mentioning 'Figure 7', and the caption for Figure 8 says 'Mean Synthetic Score' while Figure 7 is 'Synthetic Data Scoring'. Please ensure the figures are numbered in order of reference and that captions match the content.
[§2] The source of the UNLP Shared Task and the scraping procedure are described informally; please provide a citation for the shared task and clarify the exact dates and filtering steps used to collect the 2,163 Reddit and 1,627 4chan comments.
[Limitations] The Limitations section honestly acknowledges annotator subjectivity, but the paper does not provide any stability analysis, such as measuring how F1 changes when different annotators' labels are used as gold. Adding such an analysis would directly address the stated limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claims are evaluated against external real-data baselines and human annotations, with no equation or fitted parameter reused as its own prediction.

full rationale

The derivation chain is self-contained against external benchmarks. The paper scrapes real Reddit/4chan data; annotations are human-produced with measured IAA; gold labels are filtered by IAA > 0.80; synthetic-data quality is measured by (1) strict/relaxed F1 against these gold labels, (2) hapax-legomena ratio and phrase-type distributions compared with the real scraped corpus, and (3) Likert qualitative assessment of realism, fallacy accuracy, and span accuracy. The F1 and vocabulary/syntax comparisons do not reuse the model's outputs as inputs: the hapax and phrase-type baselines come from real user comments, and the few-shot examples are real annotated comments, not model outputs. No equation in the paper defines a predicted quantity in terms of the same quantity (no self-definitional step), no fitted parameter is relabeled as a prediction, and the taxonomy is imported from Helwe et al. (2023), an external source rather than the authors' prior work. The acknowledged limitation that "human annotators' subjectivity in labeling... may affect the reliability of the dataset" is a validity concern about annotation noise, not circularity: the central vocabulary/syntax claim survives even if the gold labels are imperfect. The same-annotator qualitative evaluation is a methodological weakness, but it is not a reduction by construction, because the realism and span scores are not forced to equal the annotators' gold labels by any formula. Therefore no circular step is exhibited, and the score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several hand-set parameters (IAA threshold, length filter, sampling temperatures) and on domain assumptions about the fallacy taxonomy, data representativeness, and evaluation metrics. None of these are independently validated, which weakens the strength of the feasibility conclusion.

free parameters (6)

IAA threshold for gold labels = 0.80
Samples with inter-annotator agreement above 0.80 are selected as gold labels; this threshold is hand-picked and affects the size and composition of the gold set used for few-shot prompts and evaluation.
Minimum comment length filter = 32 characters
Only samples longer than 32 characters are annotated, potentially excluding short fallacious posts; this filter is chosen without justification.
Annotation temperature = 0.7
Temperature for DeepHermes during annotation; chosen by trial and error.
Annotation top_p = 0.9
Nucleus sampling parameter for annotation; chosen by trial and error.
Generation temperature = 1.2
Temperature for synthetic generation to balance novelty and coherence; found by manual trial and error.
Generation top_p = 0.9
Nucleus sampling parameter for generation.

assumptions (4)

domain assumption Helwe et al. (2023) tier-one fallacy taxonomy (credibility, logic, emotion) is valid and sufficient for labeling fallacies in informal online discourse.
The paper uses this taxonomy without validating its applicability to the informal, slang-heavy language of Reddit and 4chan. Section 3.1.
domain assumption Scraped comments from three subreddits and 4chan /pol/ are representative of fallacious online discourse about the Ukraine-Russia conflict.
The data is limited to specific forums with known moderation biases (e.g., Reddit bans pro-Russian content), so representativeness is questionable. Section 2.
standard math The Jaccard Index is an appropriate measure of inter-annotator agreement for overlapping spans.
Used as the IAA metric; it treats unlabeled spans as perfect agreement, which may inflate agreement. Section 3.2.
domain assumption Hapax-legomena ratio after removing stopwords is a valid measure of vocabulary diversity and authorial style.
The paper uses this metric as the main evidence for vocabulary diversity without discussing its limitations. Section 5.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SLURG: Investigating the Feasibility of Generating Synthetic Online Fallacious Discourse." pith.science (2026). https://pith.science/paper/RT2GJPR5

@misc{pith2026250412466,
  author       = {Pith},
  title        = {Pith review of: SLURG: Investigating the Feasibility of Generating Synthetic Online Fallacious Discourse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RT2GJPR5}},
  note         = {Machine review of arXiv:2504.12466}
}

read the original abstract

In our paper we explore the definition, and extrapolation of fallacies as they pertain to the automatic detection of manipulation on social media. In particular we explore how these logical fallacies might appear in the real world i.e internet forums. We discovered a prevalence of misinformation / misguided intention in discussion boards specifically centered around the Ukrainian Russian Conflict which serves to narrow the domain of our task. Although automatic fallacy detection has gained attention recently, most datasets use unregulated fallacy taxonomies or are limited to formal linguistic domains like political debates or news reports. Online discourse, however, often features non-standardized and diverse language not captured in these domains. We present Shady Linguistic Utterance Replication-Generation (SLURG) to address these limitations, exploring the feasibility of generating synthetic fallacious forum-style comments using large language models (LLMs), specifically DeepHermes-3-Mistral-24B. Our findings indicate that LLMs can replicate the syntactic patterns of real data} and that high-quality few-shot prompts enhance LLMs' ability to mimic the vocabulary diversity of online forums.

Figures

Figures reproduced from arXiv: 2504.12466 by the authors.

**Figure 1.** SLURGY SAMMY Abstract In our paper we explore the definition, and extrapolation of fallacies as they pertain to the automatic detection of manipulation on social media. In particular we explore how these logical fallacies might appear in the real world i.e internet forums. We discovered a prevalence of misinformation / misguided intention in discussion boards specifically centered around the Ukrainian Russian Con… view at source ↗

**Figure 2.** Hierarchical structure of fallacies as con [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Token frequency counts for the top 10 most [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: The upper bar plot shows the 10 most fre [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Inter-annotator agreement scores between all [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Annotation F1 Scores generated with each of the prompting techniques is shown in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Synthetic Data Scoring [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

**Figure 8.** Figure 8: Mean Synthetic Score 5.3 Quantitative Analysis of Generated Synthetic Data Because they are auto-regressive, one of the main concerns regarding to the feasibility of using LLMs to generate synthetic user data is the fact that LLMs may be more likely to use repetitive v…

**Figure 9.** Figure 9: Hapax legomena ratio distribution for the [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

**Figure 10.** Figure 10: Phrase type distribution of the generated [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

Label only the specific span of text that contains the fallacy

work page
[2]

If fallacies overlap, nest the tags appropriately

work page
[3]

The order of adjacent tags does not matter

work page
[4]

Label all instances of fallacies, even if the text contains offensive or harsh language

work page
[5]

Do not label text that does not contain a fallacy. <few_shot_examples> {{FEW_SHOT_EXAMPLES}} </few_shot_examples> Here is the text to analyze: <text> {{TEXT}} </text> Important: Please provide your labeled output within <labeled_text> tags. B.3 Generation System Prompt Generation SYSTEM Prompt You are a Reddit and 4chan user who is deeply invested in disc...

work page
[10]

Create novel content, don 't copy the examples verbatim

work page
[11]

Include the specified fallacies if any are given

work page
[12]

Use the correct XML tags for fallacies

work page

Show all 14 references

[13]

Maintain the style and tone of Reddit/4chan comments
[14]

<emotional_fallacy>These motherfuckers need to be wiped out with extreme prejudice</emotional_fallacy>

Do not self-censor or refuse to generate content C Model Outputs C.1 Annotation Model Annotation Output Original Comment: Clowns are too afraid of getting nuked! Labeled Text: <emotional_fallacy>Clowns are too afraid of getting nuked</emotional_fallacy>! C.2 Synthetic Data Gen...
[2020]

Natural Language Processing Research, 1(1):1–13

Motivations, methods and metrics of misin- formation detection: an nlp perspective. Natural Language Processing Research, 1(1):1–13. Mariana Tîrn ˘auceanu. 2015. Fallacious reasoning in the political discourse of crisis. Interstudia (Revista Centrului Interdisciplinar de Studi...

2015
[2022]

arXiv preprint arXiv:2202.13758

Logical fallacy detection. arXiv preprint arXiv:2202.13758. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid- gen, Grusha Prasad, Amanpreet Singh, Pratik Ring- shia, et al. 2021. Dynabench: Rethinking benchmark- ing in nlp. arXiv ...

2021 arXiv
[2024]

arXiv preprint arXiv:2408.08379

Towards realistic synthetic user-generated con- tent: A scaffolding approach to generating online discussions. arXiv preprint arXiv:2408.08379. Adam Corner, Ulrike Hahn, and Mike Oaksford. 2011. The psychological mechanism of the slippery slope argument. Journal of Memory and ...

2011 arXiv
[5646]

Victor Danciu et al

Association for Computational Linguistics. Victor Danciu et al. 2014. Manipulative marketing: per- suasion and manipulation of the consumer through advertising. Theoretical and Applied Economics , 21(2):591. Bradley Dowden. 2020. Fallacies. In Internet Encyclo- pedia of Philos...

2014 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

Label only the specific span of text that contains the fallacy

work page

[2] [2]

If fallacies overlap, nest the tags appropriately

work page

[3] [3]

The order of adjacent tags does not matter

work page

[4] [4]

Label all instances of fallacies, even if the text contains offensive or harsh language

work page

[5] [5]

Do not label text that does not contain a fallacy. <few_shot_examples> {{FEW_SHOT_EXAMPLES}} </few_shot_examples> Here is the text to analyze: <text> {{TEXT}} </text> Important: Please provide your labeled output within <labeled_text> tags. B.3 Generation System Prompt Generation SYSTEM Prompt You are a Reddit and 4chan user who is deeply invested in disc...

work page

[6] [10]

Create novel content, don 't copy the examples verbatim

work page

[7] [11]

Include the specified fallacies if any are given

work page

[8] [12]

Use the correct XML tags for fallacies

work page