DS@GT at CheckThat! 2025: Ensemble Methods for Detection of Scientific Discourse on Social Media

Ayush Parikh; Hoang Thanh Thanh Truong; Jeanette Schofield; Maximilian Heil

REVIEW 4 major objections 6 minor 16 references

DS@GT at CheckThat! 2025: Ensemble Methods for Detection of Scientific Discourse on Social Media

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A category-routing ensemble of a fine-tuned DeBERTa and GPT-4o outperforms the baseline at detecting scientific discourse in tweets.

desk verdict A straightforward CheckThat! system description whose category-split ensemble idea is worth a look, but the reported F1 gain is a 137-tweet dev-set artifact until shown otherwise. read the letter →

arxiv 2507.06205 v1 pith:JEXBOLSO submitted 2025-07-08 cs.CL

Ayush Parikh , Hoang Thanh Thanh Truong , Jeanette Schofield , Maximilian Heil This is my paper

classification cs.CL

keywords scientificdiscoursedetectionmulti-labelclassificationensemblemodelsfinetuningfew-shotpromptingDeBERTaGPT-4osocialmedia

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the best way to detect scientific discourse in tweets is to split the work: let a fine-tuned transformer classify scientific claims and entity mentions, and let a few-shot GPT-4o decide whether the tweet references a scientific study or publication. On the competition's development set, this ensemble reaches a macro-averaged F1 of 0.86, beating the official DeBERTaV3 baseline's 0.84 and each component alone. The authors' hypothesis is that LLMs have learned citation patterns from web-scale pretraining, which makes them better at spotting references, while fine-tuned transformers are better at the two other categories. If true, the result is a practical recipe for multi-label classification on noisy social media data with imbalanced classes.

What carries the argument

The load-bearing mechanism is the category-routing ensemble: a single fine-tuned DeBERTa-base transformer, trained with a sigmoid head and binary cross-entropy for all three labels, but whose prediction for Category 2 is discarded and replaced by GPT-4o's answer. The GPT-4o prompt is constructed by retrieving the five most semantically similar tweets from the training set and presenting them as labeled examples. The transformer handles Categories 1 and 3; GPT-4o handles Category 2; the final output concatenates the two sources. The design was chosen after observing that the LLM consistently beat the transformer on Category 2 while losing on the other two.

What would settle it

Run the final ensemble and the DeBERTaV3 baseline on a fresh, separately held-out sample of tweets (or the competition's unlabeled evaluation set) and compute category-wise macro-F1. If the ensemble does not beat the baseline by a comparable margin outside the 137-tweet development set, the claimed advantage is a selection artifact.

Watch

Extended reading notes

Core claim

The central claim is that a category-routed ensemble outperforms any single model on the three-label scientific-discourse task: the fine-tuned DeBERTa-base transformer supplies the labels for 'scientific claim' and 'scientific entity mention', and GPT-4o, prompted with five semantically retrieved training tweets, supplies the label for 'reference to a scientific study or publication'. The paper reports macro-averaged F1 of 0.86 on the development set, versus the 0.84 of the DeBERTaV3 baseline and 0.85 for the best fine-tuned transformer alone. The authors interpret the large gap on the reference category (GPT-4o-mini reaches 0.90 few-shot, the transformer 0.82) as evidence that LLMs exploit citation-link structure learned from the internet. The claim is therefore that complementary strengths can be combined by category routing, and that this combination transfers to the competition setting.

Load-bearing premise

The reported score is measured on the same 137-tweet development set that was used to decide which model handles which category, so the 0.02 gain over the baseline may reflect tuning to that small set rather than general improvement.

Editorial extensions

If this is right

Systems that assign each output label to whichever model handles it best can beat both a fine-tuned transformer and a prompted LLM used alone on the same multi-label task.
Because LLMs identified scientific references well even in zero-shot mode, few-shot prompting may be a reliable way to detect study references in tweets without additional training data.
Treating the three labels as independent leaves a known dependency between references and entity mentions unused; incorporating that dependency is a stated direction for improving the pipeline.
The category-wise results on the development set identify where each approach fails (LLMs confuse claims; transformers miss references), which is directly useful for building better detectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The development set has only 137 tweets and was used to pick the ensemble design, so the reported 0.86 versus 0.84 difference is likely within the noise of the estimate; a held-out test or paired significance test would be needed to confirm the improvement.
If LLMs really are good at references because they have seen citation formats, a purely prompting-based 'reference detector' could be built and validated on other social media corpora beyond this competition's COVID-era tweets.
The category-routing idea generalizes beyond scientific discourse: any multi-label task where different base models dominate different labels could use the same ensemble scheme, with the split selected on a proper validation set.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a CLEF shared-task paper, not a methods paper. The genuinely useful bit is the breakdown showing GPT-4o beats a fine-tuned DeBERTa on Category 2 (scientific references) while DeBERTa wins on Categories 1 and 3, and the ensemble that takes the best of each. That observation is plausible, well-motivated, and consistent with what others have reported about LLMs and citation-like text. Credit where due: the paper is clearly written, the code is on GitHub, the authors explicitly acknowledge LLM non-determinism, and the error analysis (Figures 4 and 5) is honest and specific.

The soft spots are real but should not be overstated for a working-notes paper. The headline gain, 0.8611 vs 0.8375, is measured on the 137-tweet development set, and the ensemble design was chosen after inspecting that same set. No significance test, no confidence intervals, no repeated LLM runs. The internal inconsistency you noticed is telling: GPT-4o 5-shot alone gets Cat2 F1 0.89, but the combined model, which is supposed to take that prediction, reports 0.85. That gap is exactly what you would expect from run-to-run variance with one roll of the dice. Also, the intro says the team placed 6th while the abstract says 7th; minor, but it should be fixed.

My verdict: the central claim that the ensemble is an improvement over the baseline is not established beyond sampling noise. The paper itself invites this reading by calling the development set the test set (Section 3.1). But for a shared-task system description, this is within the normal range. The category-split finding is a useful data point for future work on combining encoders and LLMs. I would not cite it as evidence of a robust gain, but I would send it to peer review: a serious referee can ask the authors to re-run with fixed seeds or repeated LLM calls and to report the variance. It deserves referee time because the empirical observation about Category 2 is worth checking, and competition papers are a legitimate part of the record.

Bottom line: read it as a thoughtful competition entry with a fragile headline number. If the venue is working notes, accept with light revision; if the authors ever claim this result as a general finding, they need a proper held-out evaluation.

Referee Report

4 major / 6 minor

Summary. The paper reports DS@GT's submission to CheckThat! 2025 Task 4a (Scientific Web Discourse Detection), a multi-label classification task over tweets with three categories: scientific claims, references to scientific studies, and scientific entity mentions. The authors compare a fine-tuned DeBERTa-base transformer, zero-shot and few-shot prompting of GPT-4o mini and GPT-4o with semantic example retrieval, and a combined ensemble that uses the transformer for Categories 1 and 3 and the LLM for Category 2. On the development set (137 tweets), the ensemble is reported to achieve macro-averaged F1 0.8611, above the DeBERTaV3 baseline's 0.8375, and the paper states the team placed 7th in the competition. The code is publicly available.

Significance. The paper has practical value as a clearly described systems note: it releases code, documents prompts, and offers a falsifiable hypothesis that LLMs are disproportionately strong at detecting scientific references (Category 2). The ensemble idea is simple and reproducible, and the comparison of fine-tuned transformers versus few-shot LLMs on a domain-specific low-resource task is useful for the community. However, the empirical support for the headline improvement is weak: the result is measured on a single 137-tweet development set used also for model selection, with no error bars, significance tests, or repeated LLM runs, and Table 1 contains an internal inconsistency in the Category 2 F1 of the combined model. The significance of the paper as a claim about generalization is therefore limited until these issues are addressed.

major comments (4)

[Section 5, Section 3.1] The headline result is measured exclusively on the 137-tweet development set, and the ensemble architecture was chosen after inspecting model performance on that same set (Sections 4.1 and 4.4). Because the development set contains only 26 positive examples for Category 2, a few tweets' difference separates the reported ensemble F1 (0.86) from the baseline (0.84). No held-out validation or official evaluation-set performance is reported, and the paper explicitly says 'We will report test accuracy on this development set' (Section 3.1). The claim that the ensemble improves over the baseline is therefore not established beyond selection bias and sampling noise; reporting the official evaluation-set score from the competition would address this directly.
[Table 1, Section 4.4] The combined model is described as taking the LLM's Category 2 prediction directly, yet its reported Cat2 F1 (0.85) differs from the GPT-4o (5-shot, semantic sim) Cat2 F1 (0.89) listed in the same table. This internal inconsistency suggests either a different recipe or run-to-run variance of the non-deterministic LLM (acknowledged in Section 6.1). Please clarify which LLM output was used in the ensemble and report results over multiple runs.
[Section 5 (Table 1)] No confidence intervals, bootstrap estimates, or significance tests are reported, despite the small development set (n=137) and the acknowledged non-determinism of GPT-4o. The macro-F1 gap of 0.02 and per-category gaps of at most 0.06 are within the range that could change with one or two tweets. At minimum, the paper should provide variance over repeated LLM calls or a paired significance test (e.g., McNemar) on the development-set predictions.
[Section 5.3 / Table 1] The text states that the fine-tuned DeBERTa-base model achieved the highest F1-score in Category 1 and Category 3, but Table 1 shows the baseline achieving Cat3 F1 0.90 versus 0.87 for the fine-tuned model. This contradicts the stated rationale for using the transformer for Category 3 in the ensemble; the claim or the table should be corrected.

minor comments (6)

[Section 4.3] The reference to 'Section 3.4' should be 'Section 4.3'.
[Section 4.3] The zero-shot prompt shown duplicates the category enumeration and instruction; verify whether this repetition is intentional in the actual prompt.
[Abstract and Introduction] The abstract reports 'placed 7th in the competition,' while the Introduction reports 'ranked 6th out of 11 teams based on the macro-average F1 score'; please reconcile these statements and specify which number refers to the official evaluation set.
[Figure 4 caption, Section 6.1] The caption says GPT-4o is confused about Category 3, but the surrounding text discusses confusion for Categories 1 and 2; align the caption with the discussion.
[Table 1 / Section 5.3] The row labeled 'microsoft-deberta-v3-large' is ambiguous; clarify whether it is the pre-trained model or a fine-tuned variant, since the text discusses both.
[Throughout] The model name is written both as 'DeBERTa' and 'DeBERTA'; use one consistent spelling.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports measured development-set F1 scores from standard fine-tuning, prompting, and an ensemble; no result is defined in terms of its own inputs.

full rationale

This paper contains no formal derivation chain whose conclusion equals its premises. The central results are empirical measurements: a DeBERTa model is fine-tuned on the training split, GPT-4o is prompted in zero-shot and few-shot modes, and an ensemble is formed by taking the transformer's Category 1 and Category 3 predictions together with the LLM's Category 2 prediction. The final macro-F1 is reported on the development set, which was also used to select the model and the ensemble design. That raises a legitimate concern about optimistic evaluation (n=137, no significance testing, and Section 3.1 explicitly states 'We will report test accuracy on this development set'), but it is not circularity: the reported F1 is a measured quantity, not a quantity defined in terms of the fitted parameters, and no fitted parameter is renamed as a prediction. The ensemble architecture is a model-selection choice, not a parameter whose value is the reported score. There are no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation; references [1], [4], and [8] are used as external context or independent datasets. The competition evaluation-set ranking provides an external check on the approach. The internal Table 1 inconsistency (combined model Cat2 F1 0.85 versus GPT-4o Cat2 F1 0.89, despite the ensemble taking exactly the GPT-4o Category 2 prediction) and the dev-set-selection bias are correctness/reproducibility risks, not circular steps under the definitions used here.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the quality of the SciTweets annotations and on the representativeness of the small development set. The hyperparameters (threshold, shots, epochs, split) are hand-picked and affect the reported F1. No new entities or forces are introduced.

free parameters (4)

classification threshold = 0.5
Hand-chosen sigmoid threshold for binary labels; changing it would change the F1 scores.
number of few-shot examples = 5
Selected via SemanticSimilarityExampleSelector with FAISS; the number is set by the authors and affects LLM performance.
maximum training epochs = 20
Training used early stopping with patience 3; this schedule influences the fine-tuned DeBERTa model.
train/validation split = 90/10
Used to select the best checkpoint; the dev set is then used for final reporting, creating a selection effect.

assumptions (3)

domain assumption SciTweets annotations correctly reflect the presence of scientific claims, references, and entities in tweets.
The training and development labels are taken from this dataset and are assumed to be accurate.
domain assumption Development set performance is representative of evaluation set performance.
The paper reports results on the development set and uses them to claim improvement and rank.
standard math Macro-averaged F1 is the appropriate evaluation metric for this task.
The paper uses Equation (1) to define it, consistent with the shared task.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DS@GT at CheckThat! 2025: Ensemble Methods for Detection of Scientific Discourse on Social Media." pith.science (2026). https://pith.science/paper/JEXBOLSO

@misc{pith2026250706205,
  author       = {Pith},
  title        = {Pith review of: DS@GT at CheckThat! 2025: Ensemble Methods for Detection of Scientific Discourse on Social Media},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JEXBOLSO}},
  note         = {Machine review of arXiv:2507.06205}
}

read the original abstract

In this paper, we, as the DS@GT team for CLEF 2025 CheckThat! Task 4a Scientific Web Discourse Detection, present the methods we explored for this task. For this multiclass classification task, we determined if a tweet contained a scientific claim, a reference to a scientific study or publication, and/or mentions of scientific entities, such as a university or a scientist. We present 3 modeling approaches for this task: transformer finetuning, few-shot prompting of LLMs, and a combined ensemble model whose design was informed by earlier experiments. Our team placed 7th in the competition, achieving a macro-averaged F1 score of 0.8611, an improvement over the DeBERTaV3 baseline of 0.8375. Our code is available on Github at https://github.com/dsgt-arc/checkthat-2025-swd/tree/main/subtask-4a.

Figures

Figures reproduced from arXiv: 2507.06205 by the authors.

**Figure 2.** Label overlap in Development Dataset. To gain insights into the label distribution and potential class imbalance, we conducted an exploratory data analysis on both the training (ct_train.tsv) and development (ct_dev.tsv) datasets. Both datasets consisted of 3 columns: index, text, and labels. Each tweet was annotated with three binary labels representing a multi-label classification task: Category 1 - Scientific Cl… view at source ↗

**Figure 3.** Classification pipeline for our finalized ensemble model. The model produced raw logits, one for each category, which were then passed through a sigmoid activation to generate the probabilities of three categories. If the sigmoid probability exceeded a threshold of 0.5, the tweet was classified as belonging to that label. The model was learned by comparing its predictions to the correct labels using Binary Cross-Ent… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: GPT-4o gets confused for classifying some tweets into Category 3. 6.2. LLM Performs Well for Category 2 LLMs perform extremely well on category 2, namely determining if a tweet contains a reference to a scientific study or publication. One likely reason for this perfor…

**Figure 5.** Figure 5: GPT 4o performs well when classifying tweets into Category 2. 7. Future Work There are several ideas we could explore further in an attempt to improve model performance. We will first discuss possible improvements for model fine-tuning, followed by potential enhancemen…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 7 canonical work pages

[1]

Hafid, Y

S. Hafid, Y. S. Kartal, S. Schellhammer, K. Boland, D. Dimitrov, S. Bringay, K. Todorov, S. Dietze, Overview of the CLEF-2025 CheckThat! lab task 4 on scientific web discourse, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 - Conference and Labs of the Evaluation Forum, CLEF 2025, Madrid, Spain, 2025

work page 2025
[2]

Thorne, A

J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, FEVER: a large-scale dataset for fact extraction and VERification, in: NAACL-HLT, 2018

work page 2018
[3]

Neumann, D

M. Neumann, D. King, I. Beltagy, W. Ammar, ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing, in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019, pp. 319–327. URL: https://www. aclweb.org/anthology/W19-5034. doi:10.18653/v1/W19-5034. arXiv:arXiv:1902.07669

arXiv 2019
[4]

Hafid, S

S. Hafid, S. Schellhammer, S. Bringay, K. Todorov, S. Dietze, Scitweets - a dataset and annotation framework for detecting scientific online discourse, in: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 3988–3992. URL: https://doi.org/10...

arXiv 2022
[5]

Dimitrov, E

D. Dimitrov, E. Baran, P. Fafalios, R. Yu, X. Zhu, M. Zloch, S. Dietze, Tweetscov19 - a knowledge base of semantically annotated tweets about the covid-19 pandemic, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 2991–2998. URL: https...

arXiv 2020
[6]

TweetsKB: A Public and Large-Scale RDF Corpus of Annotated Tweets

P. Fafalios, V. Iosifidis, E. Ntoutsi, S. Dietze, TweetsKB: A public and large-scale RDF corpus of annotated tweets, CoRR abs/1810.10308 (2018). URL: http://arxiv.org/abs/1810.10308, retrieved from http://arxiv.org/abs/1810.10308

work page Pith review arXiv 2018
[7]

T. B. Brown, B. Mann, N. Ryder, et al., Language models are few-shot learners, NeurIPS (2020)

work page 2020
[8]

M. J. J. Bucher, M. Martini, Fine-tuned ‘small’ llms (still) significantly outperform zero-shot generative ai models in text classification, arXiv preprint arXiv:2406.08660 (2024). URL: https: //arxiv.org/abs/2406.08660

arXiv 2024

Show all 16 references

[9]

M. Bosley, et al., Do we still need bert in the age of gpt? comparing the benefits of domain- adaptation and in-context-learning approaches to using llms for political science research, arXiv preprint arXiv:2306.17156 (2023)

2023 arXiv
[10]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Res...

2011
[11]

Paszke, S

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-perform...

2019 arXiv
[12]

Vykopal, M

I. Vykopal, M. Pikuliak, S. Ostermann, M. Šimko, Generative large language models in automated fact-checking: A survey, 2024. URL: https://arxiv.org/abs/2407.02351. arXiv:2407.02351

2024 arXiv
[13]

Karpathy, A deep dive into llms like chatgpt, https://youtu.be/7xTGNNLPyMI?si= vLbxE2ZEvviHX02e, 2025

A. Karpathy, A deep dive into llms like chatgpt, https://youtu.be/7xTGNNLPyMI?si= vLbxE2ZEvviHX02e, 2025. YouTube video. Accessed May 2025

2025
[14]

Chase, Langchain, https://www.langchain.com/, 2022

H. Chase, Langchain, https://www.langchain.com/, 2022. Accessed May 2025

2022
[15]

Accessed May 2025

LangChain Team, Langchain - how to use chat models to call tools, https://python.langchain.com/ docs/how_to/tool_calling/, 2024. Accessed May 2025

2024
[16]

URL: http://www

PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www. pace.gatech.edu

2017

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Hafid, Y

S. Hafid, Y. S. Kartal, S. Schellhammer, K. Boland, D. Dimitrov, S. Bringay, K. Todorov, S. Dietze, Overview of the CLEF-2025 CheckThat! lab task 4 on scientific web discourse, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), Working Notes of CLEF 2025 - Conference and Labs of the Evaluation Forum, CLEF 2025, Madrid, Spain, 2025

work page 2025

[2] [2]

Thorne, A

J. Thorne, A. Vlachos, C. Christodoulopoulos, A. Mittal, FEVER: a large-scale dataset for fact extraction and VERification, in: NAACL-HLT, 2018

work page 2018

[3] [3]

Neumann, D

M. Neumann, D. King, I. Beltagy, W. Ammar, ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing, in: Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics, Florence, Italy, 2019, pp. 319–327. URL: https://www. aclweb.org/anthology/W19-5034. doi:10.18653/v1/W19-5034. arXiv:arXiv:1902.07669

arXiv 2019

[4] [4]

Hafid, S

S. Hafid, S. Schellhammer, S. Bringay, K. Todorov, S. Dietze, Scitweets - a dataset and annotation framework for detecting scientific online discourse, in: Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 3988–3992. URL: https://doi.org/10...

arXiv 2022

[5] [5]

Dimitrov, E

D. Dimitrov, E. Baran, P. Fafalios, R. Yu, X. Zhu, M. Zloch, S. Dietze, Tweetscov19 - a knowledge base of semantically annotated tweets about the covid-19 pandemic, in: Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 2991–2998. URL: https...

arXiv 2020

[6] [6]

TweetsKB: A Public and Large-Scale RDF Corpus of Annotated Tweets

P. Fafalios, V. Iosifidis, E. Ntoutsi, S. Dietze, TweetsKB: A public and large-scale RDF corpus of annotated tweets, CoRR abs/1810.10308 (2018). URL: http://arxiv.org/abs/1810.10308, retrieved from http://arxiv.org/abs/1810.10308

work page Pith review arXiv 2018

[7] [7]

T. B. Brown, B. Mann, N. Ryder, et al., Language models are few-shot learners, NeurIPS (2020)

work page 2020

[8] [8]

M. J. J. Bucher, M. Martini, Fine-tuned ‘small’ llms (still) significantly outperform zero-shot generative ai models in text classification, arXiv preprint arXiv:2406.08660 (2024). URL: https: //arxiv.org/abs/2406.08660

arXiv 2024