Pith. sign in

REVIEW 3 major objections 4 minor 21 references

GateNLP at SemEval-2025 Task 10: Hierarchical Three-Step Prompting for Multilingual Narrative Classification

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Fine-tuning LLaMA-3.2 with a three-step prompt hierarchy—domain, main narrative, sub-narrative—and synthetic data wins the English track of SemEval-2025 Task 10.

desk verdict Solid shared-task system paper with a real leaderboard result, but the hierarchy claim is not isolated from fine-tuning and synthetic data effects. read the letter →

arxiv 2505.22867 v1 pith:6VP5X7CO submitted 2025-05-28 cs.CL

classification cs.CL
keywords hierarchicalpromptingnarrativeclassificationmultilingualNLPLoRAfine-tuningsyntheticdataaugmentationSemEval-2025Task10LLaMA-3
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a structured, hierarchical prompting scheme—decide the domain first, then the main narrative, then the sub-narrative—makes a fine-tuned language model substantially better at classifying news articles into a two-level narrative taxonomy than flat or zero-shot alternatives. The specific system, H3Prompt, fine-tunes LLaMA-3.2-3B-Instruct with LoRA on translated training data plus synthetically generated articles, and then applies the three-step prompt at inference time. On the English development set it reaches 0.623 macro F1 for main narratives and 0.516 samples F1 for sub-narratives, and the submitted run placed first on the English test set among 28 teams. The broader point the authors want to establish is that decomposing a fine-grained classification task along its label hierarchy is a practical way to get strong multilingual performance without task-specific architectures.

What carries the argument

The central object is the three-step prompt hierarchy, implemented as Algorithm 1: a category check, then a main-narrative lookup using the taxonomy with explanations, then a sub-narrative lookup restricted to the chosen main narratives. The prompts ask the model to return labels as a hash-separated string, with 'Other' as an explicit fallback, so each step is a constrained generation. The hierarchy is carried by a LoRA-fine-tuned LLaMA-3.2-3B-Instruct model, and its training data are augmented by synthetic articles generated with Vicuna-7B; predictions from three models trained on different data subsets are combined by union aggregation. The authors attribute the performance gain to the structured decomposition and the synthetic-data augmentation, with the hierarchy filtering the label space at each step.

What would settle it

Train a flat, non-hierarchical LLaMA-3.2-3B-Instruct model with the same LoRA hyperparameters, translated data, synthetic data, and prompt style (one step that outputs all labels), and compare it on the same English development set; if the flat model matches or exceeds the 0.623/0.516 F1 scores, the hierarchy is not the driver of the result.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that hierarchical three-step prompting (H3Prompt) is an effective method for multilingual narrative classification: first classify each article into 'Ukraine-Russia War', 'Climate Change', or 'Other'; then, given that category, identify the main narrative(s); then, for each main narrative, assign sub-narrative(s). Each step uses a prompt that includes the relevant part of the taxonomy with explanations, and returns hash-separated labels. The system fine-tunes LLaMA-3.2-3B-Instruct with LoRA on a mix of human-annotated articles (machine-translated to English) and 8,129 synthetically generated articles, and a union-based bagging ensemble of three such models gives the best results. The authors report that this approach outperformed all baselines they tested, including zero-shot LLMs, and achieved first place on the English test set of SemEval-2025 Task 10 Subtask 2.

Load-bearing premise

The load-bearing premise is that the three-step hierarchy itself is what improves accuracy; the paper never compares against a flat fine-tuned LLaMA-3.2 model trained on the same data, so the gain could come entirely from fine-tuning and synthetic data.

Editorial extensions

If this is right

  • If H3Prompt's decomposition is the cause of the gain, then the same three-step pattern should transfer to any two-level multi-label taxonomy, not just narratives.
  • Because all non-English test articles are machine-translated to English before inference, the pipeline's multilingual success depends on translation quality, so improving translation should improve non-English scores.
  • Union-based ensembling improved recall-oriented fine F1 at some cost to precision, suggesting that for tasks where missing a narrative is costly, this aggregation is the right default.
  • Fine-grained narrative classification at this level may be good enough for fact-checking triage, letting analysts prioritize articles rather than read everything.
  • The reported 23% improvement in fine F1 from synthetic data suggests data augmentation is at least as important as the prompting hierarchy, which would change where future efforts should focus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not include a flat fine-tuned LLaMA-3.2 baseline on the same training data, so the hierarchy's specific contribution over plain fine-tuning remains untested; a flat LoRA-tuned LLaMA-3.2 with the same data might perform as well or better.
  • The reliance on English translation for all non-English input means the model never sees the original language; a system that prompts in the source language or uses multilingual LLaMA might capture language-specific framing that translation flattens.
  • The taxonomy explanations were generated by ChatGPT and manually verified; the method's transferability to new domains depends on the effort of writing those explanations, which is a hidden cost not reflected in the reported numbers.
  • Since synthetic articles are generated per sub-narrative, the augmentation may bias the model toward the generator's own writing style; evaluating on a held-out set of real articles from different outlets would test whether the gain generalizes or is an artifact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper describes GateNLP's submission to SemEval-2025 Task 10 Subtask 2, a multilingual narrative and sub-narrative classification task. The proposed method, H3Prompt, is a hierarchical three-step prompting procedure: a fine-tuned LLaMA-3.2-3B-Instruct model first classifies an article into one of two domains (Ukraine-Russia War or Climate Change), then selects main narratives, and finally assigns sub-narratives per selected main narrative. Training uses 2,091 annotated articles (non-English texts machine-translated to English) plus 8,129 synthetic articles generated by Vicuna-7B. LoRA is used for fine-tuning. On the English development set, the best single model obtains 0.577 macro F1 for coarse narratives and 0.482 samples F1 for fine sub-narratives; a union-based ensemble improves these to 0.623 and 0.516. The system achieved first place on the English test set among 28 teams, with reported ranks of 1st for Portuguese and Russian, 3rd for Bulgarian, and 5th for Hindi.

Significance. If the methodological claims are supported, the paper is a solid system description with meaningful external validation: the official SemEval leaderboard confirms that the full pipeline is competitive, and the authors supply code, hyperparameter ranges, and a synthetic-data component that is explicitly ablated. The development-set results also show a clear benefit from synthetic data (0.392 to 0.482 fine samples F1 for the H3Prompt model). However, the paper's stated contribution is the hierarchical prompting mechanism itself, and that contribution is not isolated by the presented experiments. The absence of a matched flat fine-tuned LLaMA baseline and the lack of confidence intervals leave the core attribution claim unverified. The leaderboard result validates the whole pipeline, not the specific effect of the hierarchy.

major comments (3)
  1. [Table 1 and Section 2.3] The central claim that the three-step hierarchy is responsible for the performance improvement is not isolated by the experiments. Table 1 contains no flat fine-tuned LLaMA-3.2-3B-Instruct baseline trained on the same label space, the same prompt template, and the same data in a single pass. The row 'LLaMA-3.2-3B-Instruct (B)' is a binary per-sub-narrative configuration, while the other LLaMA rows are zero-shot, so the reported differences (0.482 vs. 0.428 fine samples F1; 0.577 vs. 0.562 coarse macro F1) are consistent with gains from LoRA fine-tuning and the 8,129 synthetic articles rather than from the three-step structure. The authors should add a matched flat multi-label LLaMA-3.2 LoRA baseline to Table 1, or otherwise the hierarchy effect remains confounded.
  2. [Table 1, STD column] The reported standard deviations are large (e.g., 0.390 for the best H3Prompt fine score and 0.390 for the H3Prompt with synthetic data), yet the paper does not specify how many runs or seeds produced these values, and no confidence intervals or paired significance tests are reported. Without this information, the development-set differences that support the hierarchy claim, including the 0.482 vs. 0.428 fine-score comparison, may be within run-to-run noise. The authors should report per-seed results, the number of seeds, and ideally a paired test or confidence interval for the key comparisons.
  3. [Section 4, binary model comparison] The statement that binary classification models show 'a slight decrease in performance compared to hierarchical prompting models, reinforcing the importance of a structured three-step classification approach' is not supported by Table 1 as presented. The binary LLaMA model is not a flat multi-label classifier but a per-sub-narrative binary setup, and the binary RoBERTa model uses a different architecture and training procedure. These comparisons do not control for the hierarchy variable. A matched flat classifier sharing the H3Prompt prompt template, label set, LoRA configuration, and training data is required before making this claim.
minor comments (4)
  1. [Abstract and Section 1] There are several language errors: 'emerging as a important task' should be 'an important task', 'policy markers' should be 'policymakers', and 'We used an Vicuna LLM' should be 'a Vicuna LLM'.
  2. [Section 3 and Table 1] The paper motivates the method as multilingual, but Table 1 reports English development results only, and no per-language development scores or per-language test scores are given. Reporting the development F1 for Bulgarian, Hindi, Portuguese, and Russian would strengthen the multilingual claim and clarify where machine translation helps or hurts.
  3. [Section 2.2] The hyperparameter tuning description gives ranges and final values but does not state the selection criterion or whether the development set was used for early stopping or model selection. This is worth clarifying for reproducibility.
  4. [Algorithm 1] In Step 3, the pseudocode checks 'if n_s ∈ N_s' and otherwise assigns '(n_m, Other)', but the earlier prose says 'If no suitable sub-narrative is found, "Other" is returned.' The pseudocode could be aligned more explicitly with the prompt behavior to avoid ambiguity about whether 'Other' is a sub-narrative label or a rejection.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's central claim is an empirical result against an external shared-task benchmark, not a derivation from its own inputs.

full rationale

This is an empirical systems paper whose central claim — that the H3Prompt pipeline ranked first on the English test set of SemEval-2025 Task 10 Subtask 2 — is evaluated against an externally fixed leaderboard and organizer-provided labels, not derived from any fitted equation or self-referential definition. Table 1 reports development-set F1 scores, and Section 4 cites the test leaderboard ranks; these are measurements of an independent outcome, so no prediction is an algebraic consequence of the method's inputs. The narrative explanations in Appendix A are generated from the public taxonomy via ChatGPT and manually reviewed; they are prompt features, not the targets being predicted. Synthetic data generation uses the taxonomy's sub-narrative labels to create additional training examples, which is a data-augmentation step rather than a renamed prediction. The concern that the hierarchy's contribution is not isolated because Table 1 lacks a flat fine-tuned LLaMA baseline is an experimental-control limitation, not circularity: it weakens causal attribution but does not make the measured test performance equivalent to the training data or to the synthetic generation prompt. No load-bearing self-citation chain appears; task-specific citations are to the shared-task organizers and external technical work. No circular step can be exhibited under the required standard of showing a specific reduction of the claimed result to its own inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest on standard shared task assumptions: the gold taxonomy and annotations are correct, machine translation preserves narrative content, and synthetic data with ChatGPT explanations are faithful. Hyperparameters are tuned on the development set, which introduces a risk of overfitting, but the test leaderboard provides some external check.

free parameters (6)
  • learning_rate = 2e-4
    Tuned manually within 1e-5 to 5e-4; affects all fine-tuned models.
  • num_epochs = 5
    Tuned within 1 to 8; risk of overfitting on development set.
  • batch_size = 8
    Tuned within 2 to 16.
  • lora_alpha_and_r = 64
    Both alpha and r set to 64, tuned within 8 to 128.
  • synthetic_articles_per_subnarrative = 100
    Chosen to augment data; yields 8,129 synthetic articles.
  • synthetic_data_temperature_range = 1.0 to 1.5
    Used for diversity during synthetic article generation.
assumptions (4)
  • domain assumption The SemEval 2025 taxonomy and gold labels are correct and complete for the task.
    The paper relies entirely on the provided annotations and two-level taxonomy.
  • domain assumption The provided training and test splits are representative and uncontaminated.
    Standard shared task assumption; synthetic data generation could introduce overlap with test content if not carefully controlled.
  • domain assumption Machine translation with m2m100 preserves the narrative-relevant content of the articles.
    All non-English articles are translated to English before classification (Section 2.1).
  • domain assumption Synthetic articles generated by Vicuna and explanations generated by ChatGPT are faithful to the taxonomy and useful for training.
    The paper uses 8,129 synthetic articles; no manual quality evaluation of generated articles is reported, only manual verification of explanations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GateNLP at SemEval-2025 Task 10: Hierarchical Three-Step Prompting for Multilingual Narrative Classification." pith.science (2026). https://pith.science/paper/6VP5X7CO

@misc{pith2026250522867,
  author       = {Pith},
  title        = {Pith review of: GateNLP at SemEval-2025 Task 10: Hierarchical Three-Step Prompting for Multilingual Narrative Classification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VP5X7CO}},
  note         = {Machine review of arXiv:2505.22867}
}
read the original abstract

The proliferation of online news and the increasing spread of misinformation necessitate robust methods for automatic data analysis. Narrative classification is emerging as a important task, since identifying what is being said online is critical for fact-checkers, policy markers and other professionals working on information studies. This paper presents our approach to SemEval 2025 Task 10 Subtask 2, which aims to classify news articles into a pre-defined two-level taxonomy of main narratives and sub-narratives across multiple languages. We propose Hierarchical Three-Step Prompting (H3Prompt) for multilingual narrative classification. Our methodology follows a three-step Large Language Model (LLM) prompting strategy, where the model first categorises an article into one of two domains (Ukraine-Russia War or Climate Change), then identifies the most relevant main narratives, and finally assigns sub-narratives. Our approach secured the top position on the English test set among 28 competing teams worldwide. The code is available at https://github.com/GateNLP/H3Prompt.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 9 canonical work pages

  1. [1]

    Samy Amanatullah, Serena Balani, Angela Fraioli, Stephanie M McVicker, and Mike Gordon. 2023. Tell us how you really feel: Analyzing pro-kremlin propaganda devices & narratives to identify sentiment implications. The Propwatch Project. Illiberalism Studies Program Working Paper. https://www. illiberalism. org/tell-us-how-you-really-feel-analyzing-pro-krem...

  2. [2]

    Leo Breiman. 1996. Bagging predictors. Machine learning, 24:123--140

  3. [3]

    Travis G Coan, Constantine Boussalis, John Cook, and Mirjam O Nanko. 2021. Computer-assisted classification of contrarian claims about climate change. Scientific reports, 11(1):22320

  4. [4]

    Michael Han Daniel Han and Unsloth team. 2023. http://github.com/unslothai/unsloth Unsloth

  5. [5]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

  6. [6]

    Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1--48

  7. [7]

    Achyutarama Ganti, Eslam Ali Hassan Hussein, Steven Wilson, Zexin Ma, and Xinyan Zhao. 2023. Narrative style and the spread of health misinformation on twitter. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4266--4282

  8. [8]

    Philipp Heinrich, Andreas Blombach, Bao Minh Doan Dang, Leonardo Zilio, Linda Havenstein, Nathan Dykes, Stephanie Evert, and Fabian Sch \"a fer. 2024. Automatic identification of covid-19-related conspiracy narratives in german telegram channels and chats. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Res...

Show all 21 references
  1. [9]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

  2. [10]

    Bonka Kotseva, Irene Vianini, Nikolaos Nikolaidis, Nicol \`o Faggiani, Kristina Potapova, Caroline Gasparro, Yaniv Steiner, Jessica Scornavacche, Guillaume Jacquet, Vlad Dragu, et al. 2023. Trend analysis of covid-19 mis/disinformation narratives--a 3-year study. Plos one, 18(...

  3. [11]

    Jakub Piskorski, Tarek Mahmoud, Nikolaos Nikolaidis, Ricardo Campos, Alípio Jorge, Dimitar Dimitrov, Purificação Silvano, Roman Yangarber, Shivam Sharma, Tanmoy Chakraborty, Nuno Ricardo Guimarães, Elisa Sartori, Nicolas Stefanovitch, Zhuohan Xie, Preslav Nakov, and Giovanni D...

  4. [12]

    Jakub Piskorski, Nikolaos Nikolaidis, Nicolas Stefanovitch, Bonka Kotseva, Irene Vianini, Sopho Kharazi, Jens P Linge, et al. 2022. Exploring data augmentation for classification of climate change denial: Preliminary study. In Text2Story@ ECIR, pages 97--109

  5. [13]

    Harri Rowlands, Gaku Morio, Dylan Tanner, and Christopher D Manning. 2024. Predicting narratives of climate obstruction in social media advertising. In Findings of the Association for Computational Linguistics ACL 2024, pages 5547--5558

  6. [14]

    Shadi Shahsavari, Pavan Holur, Tianyi Wang, Timothy R Tangherlini, and Vwani Roychowdhury. 2020. Conspiracy in the time of corona: automatic detection of emerging covid-19 conspiracy theories in social media and the news. Journal of computational social science, 3(2):279--317

  7. [15]

    Nicolas Stefanovitch, Tarek Mahmoud, Nikolaos Nikolaidis, Jorge Alípio, Ricardo Campos, Dimitar Dimitrov, Purificação Silvano, Shivam Sharma, Roman Yangarber, Nuno Guimarães, Elisa Sartori, Ana Filipa Pacheco, Cecília Ortiz, Cláudia Couto, Glória Reis de Oliveira, Ari Gonçalve...

  8. [16]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

  9. [17]

    Ivan Vykopal, Mat \'u s Pikuliak, Ivan Srba, Robert Moro, Dominik Macko, and Maria Bielikova. 2023. Disinformation capabilities of large language models. arXiv preprint arXiv:2311.08838

  10. [18]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

  11. [19]

    Haiqi Zhou, David Hobson, Derek Ruths, and Andrew Piper. 2024. Large scale narrative messaging around climate change: A cross-cultural comparison. In Proceedings of the 1st Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2024), pages 143--155

  12. [20]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  13. [21]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.