{"id":"051f58be-e261-477d-8588-9e7f2a5b9559","arxiv_id":"2507.06205","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The DS@GT system achieved 0.8611 macro-F1 on the CheckThat! 2025 Task 4a development set by combining a fine-tuned DeBERTa model with GPT-4o few-shot prompting, outperforming the DeBERTaV3 baseline of 0.8375.","lead":"Our team's system for the CLEF 2025 CheckThat! Task 4a classifies tweets for scientific claims, references, and entity mentions, using a fine-tuned DeBERTa model for two categories and a GPT-4o few-shot prompt for the third. The ensemble reached a macro-averaged F1 of 0.8611 on the development set, an improvement over the DeBERTaV3 baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 0.86 vs 0.84 dev-set F1 gain is fragile: n=137, architecture chosen on the same dev set, no significance test or repeated LLM runs, and Table 1 has an internal Cat2 inconsistency.","rationale":"I read the paper as a competition system description whose main quantitative assertion is a dev-set macro-F1 improvement. The authors are transparent about what they did and provide code, which is good. The load-bearing assumption is not only that dev performance estimates evaluation performance; it is that the dev-set numbers are stable enough to support a 0.02 macro-F1 difference after the same dev set was used for architecture selection. Both concerns point in the same direction. I would not reject the paper: the system description, ablations, and qualitative analysis are useful. But the headline improvement should be read as conditional pending a statistical check. This matches the reader's CONDITIONAL verdict, so no change is needed. My emphasis differs slightly from the reader's: I focus on within-dev-set statistical fragility and LLM nondeterminism rather than on generalization to the evaluation set, hence 'partial' agreement.","tokens_in":9010,"tokens_out":4313,"duration_ms":46959,"concrete_test":"Using the released GitHub code, generate per-tweet dev-set predictions for the DeBERTaV3 baseline, the fine-tuned DeBERTa-base, GPT-4o (5-shot, semantic sim), and the combined ensemble. Compute paired bootstrap 95% confidence intervals over the 137 tweets (10,000 resamples) for the macro-F1 difference and category-wise differences, plus McNemar's test on Category 2. Repeat the GPT-4o component 5 times at fixed temperature and report mean ± std. If the ensemble-minus-baseline CI includes 0, the claimed improvement is not statistically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported improvement rests on a single 137-tweet development set, and the ensemble was chosen after inspecting that same set. In Table 1, the combined model differs from the fine-tuned DeBERTa-base only on Category 2 (F1 0.85 vs 0.82); the gap versus the baseline is 0.06 on Category 2 and 0.02 macro-F1. With only 26 positive Category 2 examples in the dev set, these gaps correspond to a handful of tweets. No confidence intervals, bootstrap, McNemar test, or repeated LLM runs are reported, and the paper itself acknowledges that LLMs are non-deterministic (Section 6.1). The internal comparison is also telling: GPT-4o (5-shot) is reported at Cat2 F1 0.89, but the combined model, which is supposed to take exactly that prediction, reports 0.85, suggesting run-to-run variance. Section 3.1 says the authors 'will report test accuracy on this development set,' confirming that no held-out evaluation is used. The central 'improvement over baseline' claim is therefore not established beyond sampling noise, regardless of whether the ensemble would generalize to the evaluation set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports DS@GT's submission to CheckThat! 2025 Task 4a (Scientific Web Discourse Detection), a multi-label classification task over tweets with three categories: scientific claims, references to scientific studies, and scientific entity mentions. The authors compare a fine-tuned DeBERTa-base transformer, zero-shot and few-shot prompting of GPT-4o mini and GPT-4o with semantic example retrieval, and a combined ensemble that uses the transformer for Categories 1 and 3 and the LLM for Category 2. On the development set (137 tweets), the ensemble is reported to achieve macro-averaged F1 0.8611, above the DeBERTaV3 baseline's 0.8375, and the paper states the team placed 7th in the competition. The code is publicly available.","tokens_in":9332,"tokens_out":6909,"duration_ms":63639,"significance":"The paper has practical value as a clearly described systems note: it releases code, documents prompts, and offers a falsifiable hypothesis that LLMs are disproportionately strong at detecting scientific references (Category 2). The ensemble idea is simple and reproducible, and the comparison of fine-tuned transformers versus few-shot LLMs on a domain-specific low-resource task is useful for the community. However, the empirical support for the headline improvement is weak: the result is measured on a single 137-tweet development set used also for model selection, with no error bars, significance tests, or repeated LLM runs, and Table 1 contains an internal inconsistency in the Category 2 F1 of the combined model. The significance of the paper as a claim about generalization is therefore limited until these issues are addressed.","major_comments":[{"comment":"The headline result is measured exclusively on the 137-tweet development set, and the ensemble architecture was chosen after inspecting model performance on that same set (Sections 4.1 and 4.4). Because the development set contains only 26 positive examples for Category 2, a few tweets' difference separates the reported ensemble F1 (0.86) from the baseline (0.84). No held-out validation or official evaluation-set performance is reported, and the paper explicitly says 'We will report test accuracy on this development set' (Section 3.1). The claim that the ensemble improves over the baseline is therefore not established beyond selection bias and sampling noise; reporting the official evaluation-set score from the competition would address this directly.","section":"Section 5, Section 3.1"},{"comment":"The combined model is described as taking the LLM's Category 2 prediction directly, yet its reported Cat2 F1 (0.85) differs from the GPT-4o (5-shot, semantic sim) Cat2 F1 (0.89) listed in the same table. This internal inconsistency suggests either a different recipe or run-to-run variance of the non-deterministic LLM (acknowledged in Section 6.1). Please clarify which LLM output was used in the ensemble and report results over multiple runs.","section":"Table 1, Section 4.4"},{"comment":"No confidence intervals, bootstrap estimates, or significance tests are reported, despite the small development set (n=137) and the acknowledged non-determinism of GPT-4o. The macro-F1 gap of 0.02 and per-category gaps of at most 0.06 are within the range that could change with one or two tweets. At minimum, the paper should provide variance over repeated LLM calls or a paired significance test (e.g., McNemar) on the development-set predictions.","section":"Section 5 (Table 1)"},{"comment":"The text states that the fine-tuned DeBERTa-base model achieved the highest F1-score in Category 1 and Category 3, but Table 1 shows the baseline achieving Cat3 F1 0.90 versus 0.87 for the fine-tuned model. This contradicts the stated rationale for using the transformer for Category 3 in the ensemble; the claim or the table should be corrected.","section":"Section 5.3 / Table 1"}],"minor_comments":[{"comment":"The reference to 'Section 3.4' should be 'Section 4.3'.","section":"Section 4.3"},{"comment":"The zero-shot prompt shown duplicates the category enumeration and instruction; verify whether this repetition is intentional in the actual prompt.","section":"Section 4.3"},{"comment":"The abstract reports 'placed 7th in the competition,' while the Introduction reports 'ranked 6th out of 11 teams based on the macro-average F1 score'; please reconcile these statements and specify which number refers to the official evaluation set.","section":"Abstract and Introduction"},{"comment":"The caption says GPT-4o is confused about Category 3, but the surrounding text discusses confusion for Categories 1 and 2; align the caption with the discussion.","section":"Figure 4 caption, Section 6.1"},{"comment":"The row labeled 'microsoft-deberta-v3-large' is ambiguous; clarify whether it is the pre-trained model or a fine-tuned variant, since the text discusses both.","section":"Table 1 / Section 5.3"},{"comment":"The model name is written both as 'DeBERTa' and 'DeBERTA'; use one consistent spelling.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"To the editor: This is a working-notes paper whose central empirical claim appears to rest on development-set performance alone. The authors should be asked to provide the official evaluation-set results if they exist, and to correct the Cat2 inconsistency in Table 1. If the reported 0.8611 is only a dev-set number, the abstract and introduction should be revised so as not to imply an evaluated test-set performance. The paper's fit for the venue is otherwise acceptable as a systems description."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a CLEF shared-task paper, not a methods paper. The genuinely useful bit is the breakdown showing GPT-4o beats a fine-tuned DeBERTa on Category 2 (scientific references) while DeBERTa wins on Categories 1 and 3, and the ensemble that takes the best of each. That observation is plausible, well-motivated, and consistent with what others have reported about LLMs and citation-like text. Credit where due: the paper is clearly written, the code is on GitHub, the authors explicitly acknowledge LLM non-determinism, and the error analysis (Figures 4 and 5) is honest and specific.\n\nThe soft spots are real but should not be overstated for a working-notes paper. The headline gain, 0.8611 vs 0.8375, is measured on the 137-tweet development set, and the ensemble design was chosen after inspecting that same set. No significance test, no confidence intervals, no repeated LLM runs. The internal inconsistency you noticed is telling: GPT-4o 5-shot alone gets Cat2 F1 0.89, but the combined model, which is supposed to take that prediction, reports 0.85. That gap is exactly what you would expect from run-to-run variance with one roll of the dice. Also, the intro says the team placed 6th while the abstract says 7th; minor, but it should be fixed.\n\nMy verdict: the central claim that the ensemble is an improvement over the baseline is not established beyond sampling noise. The paper itself invites this reading by calling the development set the test set (Section 3.1). But for a shared-task system description, this is within the normal range. The category-split finding is a useful data point for future work on combining encoders and LLMs. I would not cite it as evidence of a robust gain, but I would send it to peer review: a serious referee can ask the authors to re-run with fixed seeds or repeated LLM calls and to report the variance. It deserves referee time because the empirical observation about Category 2 is worth checking, and competition papers are a legitimate part of the record.\n\nBottom line: read it as a thoughtful competition entry with a fragile headline number. If the venue is working notes, accept with light revision; if the authors ever claim this result as a general finding, they need a proper held-out evaluation.","headline":"A straightforward CheckThat! system description whose category-split ensemble idea is worth a look, but the reported F1 gain is a 137-tweet dev-set artifact until shown otherwise.","tokens_in":9799,"tokens_out":1512,"would_cite":false,"duration_ms":20714,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A category-routing ensemble of a fine-tuned DeBERTa and GPT-4o outperforms the baseline at detecting scientific discourse in tweets.","keywords":["scientific discourse detection","multi-label classification","ensemble models","finetuning","few-shot prompting","DeBERTa","GPT-4o","social media"],"falsifier":"Run the final ensemble and the DeBERTaV3 baseline on a fresh, separately held-out sample of tweets (or the competition's unlabeled evaluation set) and compute category-wise macro-F1. If the ensemble does not beat the baseline by a comparable margin outside the 137-tweet development set, the claimed advantage is a selection artifact.","tokens_in":8768,"feed_emoji":"🔬","tokens_out":7985,"duration_ms":68564,"temperature":0.7,"pith_summary":"This paper claims that the best way to detect scientific discourse in tweets is to split the work: let a fine-tuned transformer classify scientific claims and entity mentions, and let a few-shot GPT-4o decide whether the tweet references a scientific study or publication. On the competition's development set, this ensemble reaches a macro-averaged F1 of 0.86, beating the official DeBERTaV3 baseline's 0.84 and each component alone. The authors' hypothesis is that LLMs have learned citation patterns from web-scale pretraining, which makes them better at spotting references, while fine-tuned transformers are better at the two other categories. If true, the result is a practical recipe for multi-label classification on noisy social media data with imbalanced classes.","feed_headline":"Hybrid of DeBERTa and GPT-4o beats baseline on scientific tweets","feed_subtitle":"Routing each tweet label to the model that handles it best lifts macro-F1 to 0.86 on the development set.","key_machinery":"The load-bearing mechanism is the category-routing ensemble: a single fine-tuned DeBERTa-base transformer, trained with a sigmoid head and binary cross-entropy for all three labels, but whose prediction for Category 2 is discarded and replaced by GPT-4o's answer. The GPT-4o prompt is constructed by retrieving the five most semantically similar tweets from the training set and presenting them as labeled examples. The transformer handles Categories 1 and 3; GPT-4o handles Category 2; the final output concatenates the two sources. The design was chosen after observing that the LLM consistently beat the transformer on Category 2 while losing on the other two.","core_discovery":"The central claim is that a category-routed ensemble outperforms any single model on the three-label scientific-discourse task: the fine-tuned DeBERTa-base transformer supplies the labels for 'scientific claim' and 'scientific entity mention', and GPT-4o, prompted with five semantically retrieved training tweets, supplies the label for 'reference to a scientific study or publication'. The paper reports macro-averaged F1 of 0.86 on the development set, versus the 0.84 of the DeBERTaV3 baseline and 0.85 for the best fine-tuned transformer alone. The authors interpret the large gap on the reference category (GPT-4o-mini reaches 0.90 few-shot, the transformer 0.82) as evidence that LLMs exploit citation-link structure learned from the internet. The claim is therefore that complementary strengths can be combined by category routing, and that this combination transfers to the competition setting.","pith_inferences":["The development set has only 137 tweets and was used to pick the ensemble design, so the reported 0.86 versus 0.84 difference is likely within the noise of the estimate; a held-out test or paired significance test would be needed to confirm the improvement.","If LLMs really are good at references because they have seen citation formats, a purely prompting-based 'reference detector' could be built and validated on other social media corpora beyond this competition's COVID-era tweets.","The category-routing idea generalizes beyond scientific discourse: any multi-label task where different base models dominate different labels could use the same ensemble scheme, with the split selected on a proper validation set."],"forward_implications":["Systems that assign each output label to whichever model handles it best can beat both a fine-tuned transformer and a prompted LLM used alone on the same multi-label task.","Because LLMs identified scientific references well even in zero-shot mode, few-shot prompting may be a reliable way to detect study references in tweets without additional training data.","Treating the three labels as independent leaves a known dependency between references and entity mentions unused; incorporating that dependency is a stated direction for improving the pipeline.","The category-wise results on the development set identify where each approach fails (LLMs confuse claims; transformers miss references), which is directly useful for building better detectors."],"supporting_citations":[{"why":"Defines the multi-label task, provides the training and development sets, and supplies the DeBERTaV3 baseline that the ensemble is compared against.","marker":"[1]"},{"why":"Provides the annotated tweet corpus and the three-category annotation scheme that define what counts as a claim, reference, or entity mention.","marker":"[4]"},{"why":"Establishes few-shot prompting as a method for classification, which the paper uses to get GPT-4o to label the reference category.","marker":"[7]"},{"why":"Reports that fine-tuned small models outperform zero-shot generative models on text classification, motivating the hybrid design and the paper's category-level hypothesis.","marker":"[8]"},{"why":"Supplies the semantic-similarity example selection used to choose the five training tweets included in the few-shot prompt.","marker":"[14]"}],"fun_headline_variants":["Category-routed ensemble lifts scientific tweet F1 to 0.86","DeBERTa plus GPT-4o beats baseline on scientific tweets","Ensemble routing boosts scientific discourse F1 to 0.86","How DeBERTa and GPT-4o split tweet labels for better F1","Routing tweets by label: DeBERTa and GPT-4o team up"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported score is measured on the same 137-tweet development set that was used to decide which model handles which category, so the 0.02 gain over the baseline may reflect tuning to that small set rather than general improvement.","fun_headline_variants_meta":{"raw":{"variants":["Category-routed ensemble lifts scientific tweet F1 to 0.86","DeBERTa plus GPT-4o beats baseline on scientific tweets","Ensemble routing boosts scientific discourse F1 to 0.86","How DeBERTa and GPT-4o split tweet labels for better F1","Routing tweets by label: DeBERTa and GPT-4o team up"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1352,"prompt_tokens":910,"completion_tokens":442,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":345}},"tokens_in":526,"tokens_out":442,"duration_ms":4797,"temperature":1.0,"reasoning_tokens":345,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:07:38.977078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final ensemble and the DeBERTaV3 baseline on a fresh, separately held-out sample of tweets (or the competition's unlabeled evaluation set) and compute category-wise macro-F1. If the ensemble does not beat the baseline by a comparable margin outside the 137-tweet development set, the claimed advantage is a selection artifact.","supporting_citations":[{"cited_title":"Hafid, Y","cited_arxiv_id":null,"evidence_quote":"Defines the multi-label task, provides the training and development sets, and supplies the DeBERTaV3 baseline that the ensemble is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes few-shot prompting as a method for classification, which the paper uses to get GPT-4o to label the reference category."},{"cited_title":"Chase, Langchain, https://www.langchain.com/, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-similarity example selection used to choose the five training tweets included in the few-shot prompt."}],"review_version":1}