Pith. sign in

REVIEW 3 major objections 6 minor 18 references

Team QUST at SemEval-2025 Task 10: Evaluating Large Language Models in Multiclass Multi-label Classification of News Entity Framing

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that an ensemble of instruction-tuned large language models, combined by hard voting, achieves 1st place in Hindi, 2nd in Russian, 3rd in Portuguese, 6th in Bulgarian, and 7th in English on the official test set of…

desk verdict Externally verified shared-task results with a fixable abstract error and an underspecified cross-lingual model-selection story. read the letter →

arxiv 2506.21564 v1 pith:C4YYJV6V submitted 2025-06-12 cs.CL cs.AI

Jiyan Liu , Youzheng Liu , Taihang Wang , Xiaoman Xu , Yimin Wang , Ye Jiang This is my paper
classification cs.CLcs.AI
keywords entityframingmulti-labelclassificationinstructiontuninghardvotingensemblelearningmultilingualSemEval-2025largelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper describes a system for multilingual, multi-label classification of news entity framing, where named entities in news articles are assigned fine-grained roles such as protagonist, antagonist, or innocent. The authors instruction-tune several open large language models on a small English dataset, select the three best on English development data, and combine their predictions with hard voting. In the official evaluation, the ensemble ranks first in Hindi, second in Russian, third in Portuguese, sixth in Bulgarian, and seventh in English, outperforming the official baseline substantially in every language. The central claim is that this simple recipe — English-only model selection plus majority voting — transfers surprisingly well across languages, with residual differences largely tied to training-data size.

What carries the argument

The central mechanism is a hard voting ensemble over three instruction-tuned LLMs: Phi-3-small-128k-instruct, Phi-3-medium-128k-instruct, and Phi-4. The models are fine-tuned with a directive instructing the model to assign one or more fine-grained roles to a named entity within an article; the top-3 are chosen by performance on English development data, and their output labels are combined by majority vote, with the most frequent class chosen as the final prediction.

What would settle it

Evaluate each candidate single model on the development sets of all five languages and check whether the top-3 by English score match the top-3 by each language's own score; if they differ substantially, the English-selected ensemble is suboptimal for those languages.

Watch

Extended reading notes

Core claim

On the official test set, the authors' instruction-tuned LLM ensemble with hard voting achieves Exact Match Ratio scores of 0.4684 in Hindi (rank 1), 0.5140 in Russian (rank 2), 0.4579 in Portuguese (rank 3), 0.3871 in Bulgarian (rank 6), and 0.3277 in English (rank 7), compared with baselines around 0.04–0.06. On English development data, the voting ensemble reaches 0.4725, improving by 1.1% over the best single model, Phi-4 (0.4615). The paper establishes that combining Phi-3-small, Phi-3-medium, and Phi-4 through majority voting yields consistent though modest gains, and that instruction tuning with a one-to-many directive is the main driver of the large gap over the baseline.

Load-bearing premise

The top-3 models are selected on English development data alone, and the same ensemble is applied to all languages; if the ranking of models differs across languages, the chosen ensemble may not be the best for the other languages.

Editorial extensions

If this is right

  • Instruction tuning of open LLMs provides a large improvement over a small DeBERTa baseline for entity framing across multiple languages.
  • Majority voting over the top-3 models yields consistent but modest gains over the best single model, limited by prediction similarity and task ceiling.
  • Adding development data to the training set and selecting the top-performing model by evaluation contributes to final test performance.
  • The gap between high-resource and low-resource languages in final performance appears largely attributable to training-data scale.
  • The single-label experiment shows only a 2.86% drop, indicating the dataset is heavily single-labeled yet still supports a multi-label formulation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not explored in the paper is per-language model selection on each language's own development set; if model rankings differ across languages, the English-selected ensemble may be suboptimal for non-English tracks.
  • The modest size of the single-label drop suggests that label-imbalance handling, such as threshold tuning or asymmetric loss, could squeeze additional gains without changing the model architecture.
  • The strong correlation between training-data size and final rank hints that data augmentation or cross-lingual transfer from Hindi and Russian could improve English and Bulgarian, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports the QUST team's submission to SemEval-2025 Task 10 subtask 1, a multiclass multi-label classification task that assigns fine-grained framing roles to named entities in news across five languages. The authors fine-tune several instruction-tuned LLMs on the English data, select the top-3 models based on English development results, and combine their predictions via hard voting. Official test results place the system 1st in Hindi, 2nd in Russian, 3rd in Portuguese, 6th in Bulgarian, and 7th in English, far above the shared baseline. The paper claims that instruction tuning and the voting ensemble are responsible for the strong results, supported by an English development-set comparison where voting outperforms the best single model, Phi-4, by 0.011 exact-match ratio.

Significance. If the methodological claims hold, the paper provides a useful data point on instruction tuning and ensembling for multilingual entity framing, with externally verified leaderboard positions and released code. The main strength is that the official test scores come from an external shared-task evaluation, which gives credibility to the reported rankings. The paper also releases its code, supporting reproducibility. However, the explanatory claim that voting drives the cross-lingual results is not supported by the evidence presented, because the only voting ablation is on English development data. The paper's contribution is therefore best viewed as a system description rather than a generalizable finding about ensemble methods across languages.

major comments (3)
  1. [Abstract] The abstract describes participation in SemEval-2025 Task 7 with a three-stage retrieval framework for fact-checked claim retrieval and reports 5th and 7th place finishes, but the title, body, and Section 4.3 concern SemEval-2025 Task 10 subtask 1 on entity framing. This mismatch means the abstract does not represent the paper's content and must be corrected before the paper can be considered for publication.
  2. [Sections 3.3 and 5] The description of model selection across languages is internally inconsistent. Section 3.3 states that models were compared on the English dataset and that 'the top-3 selected models for each language' are combined via hard voting, while Section 5 says the authors 'first conduct instruction tuning on large language models on the English dataset and choose the models that perform best, then adapt them to other languages' and later 'incorporate the development data into the training set and select the top-performing model.' Under one reading, per-language fine-tuning and selection occurred, which would make Table 3 irrelevant to the non-English systems; under the other, the same English-selected ensemble was applied to all languages, in which case the 0.011 exact-match improvement over Phi-4 on English development data is the only evidence for the voting benefit, and its transfer to Bulgarian, Hindi, Portuguese, and Russian is assumed without empirical support. The paper should clarify the actual procedure and provide per-language development-set comparisons of individual models versus the voting ensemble.
  3. [Section 4.3, Table 4] The official test results are reported only as final scores and ranks, with no ablation that isolates the contribution of the voting ensemble for languages other than English. The baseline scores are near 0.04-0.06, so the claim that the approach 'significantly outperforms the baseline' is not supported by any statistical test or confidence interval. Adding per-language ablations (e.g., best single model versus the ensemble) on the development sets would be necessary to support the central claim that voting, rather than fine-tuning a single strong model, is what drives the non-English results.
minor comments (6)
  1. [Section 4.2] The text states that the voting strategy 'enhances the final performance by 1.1% compared to the Phi-4,' but Table 3 shows 0.4725 versus 0.4615, which is an absolute difference of 0.011 and a relative improvement of about 2.4%. Please clarify whether '1.1%' is meant as an absolute percentage-point improvement or a relative improvement, and make the calculation consistent.
  2. [Figure 2] The caption begins with 'llustration,' which appears to be a typo for 'Illustration.'
  3. [Table 3] The last row reads 'Phi3-small+Phi-3-medium+Phi40.4725', missing spacing and a decimal separator; it should be formatted consistently with the other rows.
  4. [Table 4] The numeric formatting is inconsistent, with entries such as '0. 0403' and '0.05700'; standardize the number of decimal places and spacing.
  5. [Section 3.3] The sentence 'We first compared the performance of several baseline models' has a capitalization issue ('We first' should be 'We first'); please proofread the text throughout.
  6. [Section 3.2] The instruction strategy section only gives a one-sentence example directive and does not specify the full instruction template, hyperparameters, or training details (batch size, sequence length, hardware). Adding these details or explicitly pointing to the released code would improve reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity: the central results are externally verified leaderboard scores; the few self-citations are inspirational and do not determine the outcome.

full rationale

The paper is a SemEval system description whose main claims are the official test-set ranks in Table 4 and the English development-table comparisons in Table 3. These are measured against an external test set and a fixed development set, so there is no fitted parameter that is renamed as a prediction. The instruction-tuning and hard-voting components are standard techniques evaluated directly in Table 3; they are not justified by a chain of self-cited uniqueness theorems. The paper cites prior work by the same group (Wang et al. 2024b; Xu et al. 2024; Jiang 2023) only as inspiration for instruction tuning and voting, and those citations are not load-bearing: the 1.1% voting gain over Phi-4 is an empirical comparison on the same English development data, not a consequence of the citations. The only methodological weakness is external validity: the top-3 ensemble was chosen on English development data and then applied to Bulgarian, Hindi, Portuguese, and Russian, and Sections 1 and 5 describe the cross-lingual training protocol inconsistently. That is an under-support issue about generalization, not a circular-reasoning issue, because the test-set ranks are externally determined. The abstract's reference to Task 7 and retrieval is an apparent copy-paste artifact and does not affect the derivation chain.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The paper has no invented entities and relies on standard ML hyperparameters (learning rate, epochs) and domain assumptions about the shared task data and cross-lingual transfer. The cross-lingual transfer assumption is load-bearing but unexamined.

free parameters (3)
  • learning rate = 1e-5 for Qwen2-7B, 1e-4 for other models
    Chosen by hand without systematic search (Section 3.3). These values affect fine-tuning outcomes and are not fitted to the test set.
  • number of training epochs = 10 or 20
    Two schemes selected based on computational efficiency; the exact value per language is not specified (Section 3.3).
  • instruction template = hand-crafted version
    The exact instruction text is given as an example, but the final used version is not fully specified (Section 3.2).
assumptions (2)
  • domain assumption The SemEval-2025 Task 10 dataset and annotations are correct, and the official evaluation metric is exact match ratio.
    The paper relies entirely on the shared task organizers for data and scoring (Section 3.1, Section 4.3).
  • domain assumption Instruction-tuned LLMs fine-tuned on English transfer to other languages without per-language model selection.
    The paper selects models on English development data and applies them to Bulgarian, Hindi, Portuguese, and Russian (Section 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Team QUST at SemEval-2025 Task 10: Evaluating Large Language Models in Multiclass Multi-label Classification of News Entity Framing." pith.science (2026). https://pith.science/paper/C4YYJV6V

@misc{pith2026250621564,
  author       = {Pith},
  title        = {Pith review of: Team QUST at SemEval-2025 Task 10: Evaluating Large Language Models in Multiclass Multi-label Classification of News Entity Framing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C4YYJV6V}},
  note         = {Machine review of arXiv:2506.21564}
}
read the original abstract

This paper describes the participation of QUST_NLP in the SemEval-2025 Task 7. We propose a three-stage retrieval framework specifically designed for fact-checked claim retrieval. Initially, we evaluate the performance of several retrieval models and select the one that yields the best results for candidate retrieval. Next, we employ multiple re-ranking models to enhance the candidate results, with each model selecting the Top-10 outcomes. In the final stage, we utilize weighted voting to determine the final retrieval outcomes. Our approach achieved 5th place in the monolingual track and 7th place in the crosslingual track. We release our system code at: https://github.com/warmth27/SemEval2025_Task7.

Figures

Figures reproduced from arXiv: 2506.21564 by the authors.

Figure 1
Figure 1. Distribution of the number of labels in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. llustration of the overall workflow in this paper. "Model 1", "Model 2" and "Model 3" represent the top-3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 4 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024 a . Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024 b . Phi-4 technical report. arXiv preprint arXiv:2412.08905

    arXiv 2024

  3. [3]

    Abdul Ahad Abro. 2021. Vote-based: Ensemble approach. Sakarya University Journal of Science, 25(3):858--866

    work page 2021

  4. [4]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    arXiv 2024

  5. [5]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. http://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    arXiv 2021

  6. [6]

    Hanan Ghali Jabbar. 2024. Advanced threat detection using soft and hard voting techniques in ensemble learning. Journal of Robotics and Control (JRC), 5(4):1104--1116

    work page 2024

  7. [7]

    Ye Jiang. 2023. Team qust at semeval-2023 task 3: A comprehensive study of monolingual and multilingual approaches for detecting online news genre, framing and persuasion techniques. arXiv preprint arXiv:2304.04190

    arXiv 2023

  8. [8]

    Tarek Mahmoud, Zhuohan Xie, Dimitar Dimitrov, Nikolaos Nikolaidis, Purifica c \ a o Silvano, Roman Yangarber, Shivam Sharma, Elisa Sartori, Nicolas Stefanovitch, Giovanni Da San Martino, et al. 2025. Entity framing and role portrayal in the news. arXiv preprint arXiv:2502.14718

    arXiv 2025

Show all 18 references
  1. [9]

    Jakub Piskorski, Tarek Mahmoud, Nikolaos Nikolaidis, Ricardo Campos, Alípio Jorge, Dimitar Dimitrov, Purificação Silvano, Roman Yangarber, Shivam Sharma, Tanmoy Chakraborty, Nuno Ricardo Guimarães, Elisa Sartori, Nicolas Stefanovitch, Zhuohan Xie, Preslav Nakov, and Giovanni D...

    2025

  2. [10]

    Yulei Qin, Yuncheng Yang, Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. 2024. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. arXiv preprint arXiv:2...

    2024 arXiv

  3. [11]

    Nicolas Stefanovitch, Tarek Mahmoud, Nikolaos Nikolaidis, Jorge Alípio, Ricardo Campos, Dimitar Dimitrov, Purificação Silvano, Shivam Sharma, Roman Yangarber, Nuno Guimarães, Elisa Sartori, Ana Filipa Pacheco, Cecília Ortiz, Cláudia Couto, Glória Reis de Oliveira, Ari Gonçalve...

    2025

  4. [12]

    Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. 2024 a . A survey on data selection for llm instruction tuning. arXiv preprint arXiv:2402.05123

    2024 arXiv

  5. [13]

    Taihang Wang, Xiaoman Xu, Yimin Wang, and Ye Jiang. 2024 b . Instruction tuning vs. in-context learning: revisiting large language models in few-shot computational social science. arXiv preprint arXiv:2409.14673

    2024 arXiv

  6. [14]

    Xiaoman Xu, Xiangrun Li, Taihang Wang, Jianxiang Tian, and Ye Jiang. 2024. Team qust at semeval-2024 task 8: A comprehensive study of monolingual and multilingual approaches for detecting ai-generated text. arXiv preprint arXiv:2402.11934

    2024 arXiv

  7. [15]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  8. [16]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792

    2023

  9. [17]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  10. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.