{"id":"fff473ee-fa7b-4473-8191-337a5c055f54","arxiv_id":"2504.17974","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PolyHope V2 adds sarcasm labels to existing English and Spanish hope-speech corpora and shows fine-tuned transformers outperform zero-shot and few-shot LLMs.","lead":"This paper introduces PolyHope V2, a bilingual English-Spanish dataset of about 30,000 tweets labeled for four kinds of hope speech, including sarcastic hope. It reports that fine-tuned transformer models beat large language models like GPT-4 and Llama 3 on this task.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sarcasm gold labels are generated by the authors' own transformers (English) and by GPT-4o translation (Spanish), so the reported transformer advantage on sarcasm may measure recovery of machine-generated labels rather than human-labeled sarcasm.","rationale":"I read the paper as a dataset-plus-benchmark contribution whose central claim has two parts: the resource is a valid fine-grained hope dataset, and fine-tuned transformers beat LLMs on it. The reader's conditional verdict targets the right part: the sarcasm labels are the novel component and they are not independently human-labeled. I agree with that assessment. The pipeline in Section 3.3 creates a circularity for the multiclass comparison: English Sarcasm labels are selected by the same families of classifiers being benchmarked, and Spanish Sarcasm is GPT-4o's own translation, so the model-vs-LLM gap on Sarcasm could be a property of the label-generation process. Separately, Table 3 does not reconcile with the stated binary merge: English multiclass Not Hope (4,081) plus Sarcasm (1,259) is 5,340, not the binary Not Hope 5,081, and the same 259 gap appears in Spanish; this is an internal data-quality red flag but not the central claim's weakest assumption. The transformer-vs-LLM comparison on the three hope subtypes, which come from prior human-annotated corpora, is likely unaffected. The independent-annotation test would settle whether the sarcasm concern lands. If it lands, the paper should restrict its claims to classifier-consistent labels or re-annotate; if it does not, the conditional verdict can be upgraded.","tokens_in":14365,"tokens_out":7733,"duration_ms":77502,"concrete_test":"Independently re-annotate a stratified random sample of 300 Sarcasm-labeled tweets (150 English, 150 Spanish) using the paper's category definitions, with two fresh annotators blind to the published labels; compute pairwise Cohen's kappa and exact agreement with PolyHope V2. If kappa is below 0.6 or agreement with the published Sarcasm labels is below 75%, the sarcasm benchmark results are not reliable evidence about human-labeled sarcasm and the headline claim needs to be restricted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 describes the novel Sarcasm class as constructed, not human-annotated. For English, tweets from iSarcasmEval and Sarcasm Corpus V2 were filtered by the authors' hope classifier and sarcasm classifier, with no stated human verification. For Spanish, the 1,259 Sarcasm instances were produced by translating the English instances with GPT-4o and verified only by two annotators checking tone. The paper's central claim is that fine-tuned transformers 'especially' outperform prompt-based LLMs in distinguishing sarcasm; if the gold labels are outputs of the same transformer family that is fine-tuned, then high sarcasm recall may reflect learning the label generator's decision boundary, while GPT-4 and Llama-3 are scored against labels they did not generate. The claim about a reliable sarcasm-labeled hope-speech resource is therefore not yet supported for human sarcasm. This is load-bearing because Sarcasm is the main novel contribution; the hope-subtype labels derive from prior human-annotated PolyHope corpora and are less exposed to this concern.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces PolyHope V2, a bilingual (English/Spanish) tweet dataset for fine-grained hope speech detection with four classes: Generalized, Realistic, Unrealistic, and Sarcastic hope. The English part derives from the existing PolyHope corpus, extended with sarcasm instances selected from iSarcasmEval and Sarcasm Corpus V2 using the authors' own hope and sarcasm classifiers; the Spanish part is based on the MindHope corpus, with sarcasm instances produced by GPT-4o translation of English sarcasm tweets. The authors benchmark fine-tuned transformers (RoBERTa, ALBERT, ELECTRA, DistilBERT) against GPT-4 and Llama-3 in zero-shot and few-shot settings, reporting that fine-tuned transformers consistently outperform prompt-based LLMs, especially for sarcasm. The paper also includes qualitative error analysis and confusion matrices for the best models.","tokens_in":14496,"tokens_out":7200,"duration_ms":57133,"significance":"If the dataset and benchmark are valid, PolyHope V2 would fill a gap by adding a sarcasm dimension to hope speech detection in two languages and by providing a systematic comparison of fine-tuned vs. prompt-based models on this task. The evaluation is standard (5-fold cross-validation, multiple metric families) and the paper is generally transparent about construction choices. However, the central novel component—the sarcasm labels—is generated by the authors' own classifiers (English) and by machine translation (Spanish), with limited human verification. Until the reliability of those labels is established, the reported superiority of fine-tuned transformers on sarcasm, and the dataset itself as a benchmark, are not fully supported. The internal inconsistency in the Table 3 class counts further weakens confidence in the resource.","major_comments":[{"comment":"The dataset statistics in Table 3 are internally inconsistent. For English, the binary Hope count is 4,434, whereas the sum of the three hope subtypes (Generalized 2,335 + Realistic 982 + Unrealistic 858) is 4,175, and the binary Not-Hope count is 5,081, whereas the sum of multiclass Not-Hope and Sarcasm (4,081 + 1,259) is 5,340. The same 259-instance offset appears in Spanish (binary Hope 9,654 vs. subtype sum 9,395; binary Not-Hope 10,788 vs. multiclass Not-Hope + Sarcasm 11,047). Since Section 3.3 states that sarcasm is merged into Not-Hope in the binary setting, these numbers cannot both be correct. This inconsistency undermines the dataset description and must be resolved before the corpus statistics can be used.","section":"Table 3"},{"comment":"The English Sarcasm class is constructed by filtering iSarcasmEval and Sarcasm Corpus V2 with the authors' own Hope and Sarcasm classifiers from [9], and the paper reports no human verification of these labels. The same transformer architectures are then fine-tuned and evaluated on this data (Tables 6 and 8), creating a circularity: the 94%+ sarcasm recall reported in Section 6 may reflect the models' ability to reproduce the decision boundary of the label generator rather than to recognize human-annotated sarcasm. Since sarcasm is the paper's main novel contribution, the claim that the dataset contains reliable sarcasm labels is not supported. Please provide evidence of label quality, such as human agreement on a sample, or temper the claims accordingly.","section":"Section 3.3"},{"comment":"The Spanish Sarcasm instances are translations of the English instances produced by GPT-4o, with only two annotators verifying 'tone.' This does not establish that the Spanish items are natural expressions of sarcasm in Spanish social media, and the protocol for adjudicating tone is not described. The paper's claim of a 'multilingual' sarcasm resource is therefore overstated; the Spanish sarcasm subset is a machine-translated artifact. Please report the verification protocol and inter-annotator agreement, and consider whether the multilingual claim can be sustained.","section":"Section 3.3"},{"comment":"The few-shot evaluation is underspecified. Section 4.2.1 says '5 balanced examples per class,' but Section 4.2.2 says 'a sample of 10 per label' and Section 5 refers to 'ten in-context demonstrations.' It is also not stated whether the demonstration examples are drawn from the training folds or from the full dataset. If test instances are included among the demonstrations, the reported few-shot scores would be inflated. Please clarify the exact sampling procedure and the number of demonstrations, and ensure no test leakage.","section":"Section 4.2"}],"minor_comments":[{"comment":"The paragraph reports both 'roughly 33,300 unique' tweets after preprocessing and 'only 35,000 tweets were left'; these numbers are inconsistent and should be reconciled.","section":"Section 3.2"},{"comment":"The few-shot example count is given as '5 balanced examples per class' in Section 4.2.1, '10 per label' in Section 4.2.2, and 'ten in-context demonstrations' in Section 5; please unify the description.","section":"Section 4.2"},{"comment":"There is a typo 'RoBER T a' in the paragraph after Table 8; it should read 'RoBERTa'.","section":"Section 5"},{"comment":"Several reference entries contain placeholder '???' (e.g., [6], [13]), which should be replaced with complete publication details.","section":"References"},{"comment":"The abstract and introduction state 'over 30,000 annotated tweets,' but the sum of the English and Spanish totals (9,515 + 20,442 = 29,957) is just under 30,000; please adjust the wording.","section":"Abstract / Section 1"},{"comment":"The confusion matrices referenced as Figures 1 and 2 are not visible in the submitted text; please ensure they are included and legible in the final version.","section":"Figures 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern about the English sarcasm labels is the main risk to the paper's contribution. If the authors can provide human-annotated validation of a sample, or obtain human labels through the associated shared task, the claims would be substantially strengthened. I also recommend correcting the Table 3 counts and clarifying the few-shot protocol before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is PolyHope V2 as an assembled bilingual resource: 30k tweets, four hope subtypes, public release, and a clean benchmark of fine-tuned transformers versus prompt-based LLMs. The error analysis is genuinely informative—the confusion patterns between Generalized, Realistic, and Unrealistic Hope tell you something real about the task. For the hope-subtype labels, which come from prior human-annotated corpora, the empirical claim that fine-tuned transformers beat few-shot GPT-4 and Llama-3 is plausible and probably holds up.\n\nThe soft spots are in the sarcasm layer, which is the paper's main new contribution. Section 3.3 says the English sarcasm instances were generated by filtering existing sarcasm datasets with the authors' own hope and sarcasm classifiers, with no stated human verification. The Spanish sarcasm instances are GPT-4o translations of those English instances, checked by two annotators for tone only. When the gold labels are produced by the same classifier families being fine-tuned, the reported 94%+ sarcasm recall may largely reflect the model learning its own label generator's decision boundary. That is load-bearing for the paper's most distinctive claim.\n\nThere is also a concrete arithmetic problem in Table 3. The binary Hope counts do not match the sum of the three hope subtypes in either language (English 4,434 vs. 4,175; Spanish 9,654 vs. 9,395), and binary Not-Hope doesn't match Not-Hope plus Sarcasm in multiclass. For a dataset paper, that undercuts confidence even where the underlying labels are fine. Minor inconsistency: the few-shot prompt description says 5 balanced examples per class, while Section 4.2.2 says 10 per label.\n\nThe central comparison on hope subtypes is not invalidated by the sarcasm label issue, and the paper is honest enough to discuss limitations in the qualitative analysis. But as a resource paper, PolyHope V2 needs the sarcasm labels re-derived or at least independently human-annotated with reported IAA, and Table 3 fixed. This deserves a serious referee because the resource and benchmark could be useful, but I would not accept the sarcasm claims as they stand. I'd cite the hope-subtype benchmark once the dataset is cleaned, not the sarcasm layer.","headline":"The hope-subtype benchmark is probably fine, but the paper's main novelty—the sarcasm layer—is built from the authors' own classifiers and machine translation, so the headline claims about sarcasm need independent human annotation before this is a reliable resource.","tokens_in":15104,"tokens_out":1691,"would_cite":false,"duration_ms":18526,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PolyHope V2, a corpus of 30,000 English and Spanish tweets labeled for four hope subtypes, shows that fine-tuned transformers outperform large language models at detecting hope and sarcasm, and that the hardest remaining errors lie among…","keywords":["Hope Speech Detection","Sarcasm Detection","Multilingual NLP","Emotion Recognition","Fine-grained Sentiment Analysis","Transformer Fine-tuning","Large Language Models","English and Spanish"],"falsifier":"Take a random sample of 500 English and 500 Spanish tweets labeled Sarcasm, have independent human annotators label them for sarcasm without seeing the paper's labels, and measure agreement; if agreement is at chance level, or if models retrained on the human labels score below the reported 94 percent sarcasm recall, the reported advantage and benchmark comparisons rest on unreliable ground truth.","tokens_in":14125,"feed_emoji":"💬","tokens_out":5401,"duration_ms":51946,"temperature":0.7,"pith_summary":"Hope on social media is not a single emotion, this paper argues: it splits into generalized optimism, realistic expectation, unrealistic wishful thinking, and outright sarcasm. PolyHope V2 is a bilingual corpus of 9,515 English and 20,442 Spanish tweets labeled with this four-way taxonomy plus an explicit sarcasm class, and the paper's central claim is that fine-tuning pretrained transformers on this corpus detects these subtypes substantially better than prompting large language models like GPT-4 and Llama 3 even with examples. In binary hope-versus-not-hope terms the best English transformer reaches 86.5 percent macro F1, and in the harder five-class setting it stays near 79 percent weighted F1. If the labels hold, the result matters because systems used in education, mental-health monitoring, and content moderation would be able to tell genuine encouragement from wishful thinking or mockery in two languages.","feed_headline":"Transformers beat GPT-4 at spotting hope and sarcasm in tweets","feed_subtitle":"A 30,000-tweet bilingual corpus labels generalized, realistic, unrealistic, and sarcastic hope; fine-tuning wins.","key_machinery":"The load-bearing object is the annotated dataset itself: roughly 30,000 tweets with a four-way hope taxonomy (Generalized, Realistic, Unrealistic, Sarcastic), built from the earlier English PolyHope corpus, a Spanish counterpart collected with translated hope trigger words, and a sarcasm layer mined from two existing sarcasm datasets plus GPT-4o-translated Spanish sarcasm. The paper's quantitative findings all flow from pairing this annotation schema with supervised fine-tuning of four transformer architectures under stratified 5-fold cross-validation; the fine-tuning procedure, token-level training, and fixed label vocabulary are what the paper credits for beating prompt-based LLMs.","core_discovery":"The paper's discovery is that explicitly labeling sarcasm within hope speech changes the task: sarcastic tweets behave like not-hope in binary settings, and keeping sarcasm as a separate fifth class lets models learn the pragmatic cues that distinguish ironic hope from sincere hope. Across English and Spanish, fine-tuned transformers (RoBERTa for English, RoBERTa or Albert for Spanish) deliver the highest and most class-balanced scores, around 86 percent macro F1 in binary settings and 72 to 76 percent macro F1 in multiclass settings, while GPT-4 and Llama 3 in zero- and few-shot modes fall 8 to 9 weighted-F1 points behind even with demonstrations and collapse to 36 to 40 percent macro F1 in zero-shot multiclass. The confusion matrices show the remaining errors are not between hope and sarcasm but among the three sincere hope subtypes, especially realistic versus generalized hope.","pith_inferences":["If the labels hold up, sarcasm-aware hope detection could transfer to other languages by translating the taxonomy rather than retraining from scratch, a direction the paper does not develop.","Adding temporal or evidential features (futurity markers, probability adverbs, plausibility knowledge) could push realistic-versus-generalized accuracy above the confusion levels the paper reports.","The four-way distinction maps naturally onto psychological hope constructs, which could make the classifier useful beyond social media, for example in mental-health screening where unrealistic hope may signal a different state than grounded optimism.","A single prompt template may understate LLM ability; prompt-optimized runs could close part of the gap, a question the paper leaves open."],"forward_implications":["Binary hope detection improves when sarcastic tweets are merged with not-hope, sharpening the decision boundary the classifier must learn.","Multiclass accuracy drops by about ten to twelve points relative to binary, and the hardest errors are among the three sincere hope subtypes, not between hope and sarcasm.","Sarcasm is detected at over 94 percent recall by the best fine-tuned transformer in both languages, suggesting the pragmatic cues are learnable from the annotation layer.","Prompt-based LLMs lag by 8 to 9 weighted-F1 points even with ten demonstrations and by far more in zero-shot multiclass, so fine-tuning remains the stronger route for this task."],"supporting_citations":[{"why":"Supplies the original English PolyHope corpus, its annotation guidelines, and the best transformer model whose fine-tuned variant filters English sarcasm.","marker":"[9]"},{"why":"Supplies the Spanish PolyHope corpus used for the Spanish binary and multiclass splits.","marker":"[40]"},{"why":"One of the two benchmark sarcasm datasets mined for English sarcastic tweets relabeled as hope.","marker":"[41]"},{"why":"The other English sarcasm source used to expand the sarcasm class in PolyHope V2.","marker":"[42]"},{"why":"Defines the GPT-4 model benchmarked in zero-shot and few-shot conditions.","marker":"[43]"},{"why":"Defines the Llama 3 model benchmarked as the open-source LLM comparison.","marker":"[45]"}],"fun_headline_variants":["Fine-tuned transformers beat GPT-4 and Llama 3 on hope speech","Sarcastic hope detection: fine-tuned transformers outperform LLMs","PolyHope V2: 30k tweets, four hope types, transformers lead","Bilingual hope speech: transformers trump prompt-based GPT-4"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sarcasm labels are treated as ground truth even though they were generated by the authors' own classifiers (English) and by GPT-4o translation with only light expert verification (Spanish), so any bias in those generators would inflate the reported sarcasm performance and the comparisons built on it.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned transformers beat GPT-4 and Llama 3 on hope speech","Sarcastic hope detection: fine-tuned transformers outperform LLMs","PolyHope V2: 30k tweets, four hope types, transformers lead","Bilingual hope speech: transformers trump prompt-based GPT-4"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2444,"prompt_tokens":930,"completion_tokens":1514,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1434}},"tokens_in":546,"tokens_out":1514,"duration_ms":13742,"temperature":1.0,"reasoning_tokens":1434,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:26:58.036896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 500 English and 500 Spanish tweets labeled Sarcasm, have independent human annotators label them for sarcasm without seeing the paper's labels, and measure agreement; if agreement is at chance level, or if models retrained on the human labels score below the reported 94 percent sarcasm recall, the reported advantage and benchmark comparisons rest on unreliable ground truth.","supporting_citations":[{"cited_title":"Expert Systems with Applications 225, 120078 (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the original English PolyHope corpus, its annotation guidelines, and the best transformer model whose fine-tuned variant filters English sarcasm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Spanish PolyHope corpus used for the Spanish binary and multiclass splits."},{"cited_title":"In: The 16th International Workshop on Semantic Evaluation 2022, pp","cited_arxiv_id":null,"evidence_quote":"One of the two benchmark sarcasm datasets mined for English sarcastic tweets relabeled as hope."},{"cited_title":"In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp","cited_arxiv_id":null,"evidence_quote":"The other English sarcasm source used to expand the sarcasm class in PolyHope V2."}],"review_version":1}