{"id":"38028c9a-66c1-40ae-8cc8-5a2f4299aa13","arxiv_id":"2506.21564","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An instruction-tuned LLM ensemble with hard voting achieves top ranks in multilingual entity framing, ranking 1st in Hindi, 2nd in Russian, 3rd in Portuguese in SemEval-2025 Task 10.","lead":"This paper describes a system for SemEval-2025 Task 10, assigning fine-grained roles to named entities in multilingual news, using instruction-tuned LLMs and hard voting. It ranked first in Hindi and second in Russian, showing that LLM fine-tuning plus ensemble voting works across languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Voting-ensemble benefit is established only on English development data; cross-lingual applicability and per-language model-selection procedure are ambiguous, so the claimed cross-lingual advantage of the ensemble is under-supported.","rationale":"The paper's headline empirical result is an official SemEval leaderboard placement, so it is externally verifiable and I do not challenge it. The reviewer's conditional verdict is driven mainly by the unrelated abstract, which is a genuine editorial problem, but it does not weaken the body's official results. The most load-bearing scientific concern is about the internal logic of the method explanation: the ensemble is selected on English development data and then claimed to help across five languages, while the text is inconsistent about whether per-language fine-tuning or model selection happened. This matters because the paper's contribution is not just 'we ranked 1st' but 'this voting ensemble is why' (Sections 4.2 and 5). If the submitted systems were actually selected or tuned per language, the English Table 3 comparison is not evidence for the non-English submissions. If they were English-only transfer systems, the cross-lingual transfer of the exact model ranking is assumed, not tested. Either way, the causal claim for voting is under-supported. A concrete reproduction on each language's development set would settle it. The verdict remains CONDITIONAL, unchanged from the reader, because the issues are addressable with additional experiments or clarifications and do not overturn the verified leaderboard result.","tokens_in":5036,"tokens_out":7422,"duration_ms":84959,"concrete_test":"Using the released repository and official datasets, fine-tune the seven candidate models under the paper's stated settings, evaluate on each language's development set, and compare the per-language top-3 with the English top-3 of Phi-3-small, Phi-3-medium, and Phi-4. If any non-English language has a different top-3, or if applying the English-selected ensemble to that language's development set changes exact match ratio by more than 0.011 relative to the best single model, then the paper's stated selection procedure does not establish the ensemble's cross-lingual benefit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The official leaderboard positions in Table 4 are externally verified and are not in question. The load-bearing part of the paper's explanatory claim is that the hard-voting ensemble of Phi-3-small, Phi-3-medium, and Phi-4 produced those results. That ensemble was chosen solely from English development data (Section 4.2, Table 3), where it improved over Phi-4 by 0.011 in exact match ratio. The paper never shows that this ranking or this 0.011 gain holds for Bulgarian, Hindi, Portuguese, or Russian. Section 1 states that models were fine-tuned in English and then applied to other languages, while Section 3.3 says top-3 models were selected 'for each language' and Section 5 says development data were added to training before selecting the top-performing model. These descriptions are mutually inconsistent. Under one reading, per-language fine-tuning and selection occurred, making Table 3 irrelevant to the submitted non-English systems; under the other reading, the ensemble's cross-lingual advantage rests entirely on the unverified assumption that English development ranking transfers across languages with very different data sizes and language families. In either case, the paper does not support the claim that voting is what drives the non-English results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the QUST team's submission to SemEval-2025 Task 10 subtask 1, a multiclass multi-label classification task that assigns fine-grained framing roles to named entities in news across five languages. The authors fine-tune several instruction-tuned LLMs on the English data, select the top-3 models based on English development results, and combine their predictions via hard voting. Official test results place the system 1st in Hindi, 2nd in Russian, 3rd in Portuguese, 6th in Bulgarian, and 7th in English, far above the shared baseline. The paper claims that instruction tuning and the voting ensemble are responsible for the strong results, supported by an English development-set comparison where voting outperforms the best single model, Phi-4, by 0.011 exact-match ratio.","tokens_in":5284,"tokens_out":3610,"duration_ms":39983,"significance":"If the methodological claims hold, the paper provides a useful data point on instruction tuning and ensembling for multilingual entity framing, with externally verified leaderboard positions and released code. The main strength is that the official test scores come from an external shared-task evaluation, which gives credibility to the reported rankings. The paper also releases its code, supporting reproducibility. However, the explanatory claim that voting drives the cross-lingual results is not supported by the evidence presented, because the only voting ablation is on English development data. The paper's contribution is therefore best viewed as a system description rather than a generalizable finding about ensemble methods across languages.","major_comments":[{"comment":"The abstract describes participation in SemEval-2025 Task 7 with a three-stage retrieval framework for fact-checked claim retrieval and reports 5th and 7th place finishes, but the title, body, and Section 4.3 concern SemEval-2025 Task 10 subtask 1 on entity framing. This mismatch means the abstract does not represent the paper's content and must be corrected before the paper can be considered for publication.","section":"Abstract"},{"comment":"The description of model selection across languages is internally inconsistent. Section 3.3 states that models were compared on the English dataset and that 'the top-3 selected models for each language' are combined via hard voting, while Section 5 says the authors 'first conduct instruction tuning on large language models on the English dataset and choose the models that perform best, then adapt them to other languages' and later 'incorporate the development data into the training set and select the top-performing model.' Under one reading, per-language fine-tuning and selection occurred, which would make Table 3 irrelevant to the non-English systems; under the other, the same English-selected ensemble was applied to all languages, in which case the 0.011 exact-match improvement over Phi-4 on English development data is the only evidence for the voting benefit, and its transfer to Bulgarian, Hindi, Portuguese, and Russian is assumed without empirical support. The paper should clarify the actual procedure and provide per-language development-set comparisons of individual models versus the voting ensemble.","section":"Sections 3.3 and 5"},{"comment":"The official test results are reported only as final scores and ranks, with no ablation that isolates the contribution of the voting ensemble for languages other than English. The baseline scores are near 0.04-0.06, so the claim that the approach 'significantly outperforms the baseline' is not supported by any statistical test or confidence interval. Adding per-language ablations (e.g., best single model versus the ensemble) on the development sets would be necessary to support the central claim that voting, rather than fine-tuning a single strong model, is what drives the non-English results.","section":"Section 4.3, Table 4"}],"minor_comments":[{"comment":"The text states that the voting strategy 'enhances the final performance by 1.1% compared to the Phi-4,' but Table 3 shows 0.4725 versus 0.4615, which is an absolute difference of 0.011 and a relative improvement of about 2.4%. Please clarify whether '1.1%' is meant as an absolute percentage-point improvement or a relative improvement, and make the calculation consistent.","section":"Section 4.2"},{"comment":"The caption begins with 'llustration,' which appears to be a typo for 'Illustration.'","section":"Figure 2"},{"comment":"The last row reads 'Phi3-small+Phi-3-medium+Phi40.4725', missing spacing and a decimal separator; it should be formatted consistently with the other rows.","section":"Table 3"},{"comment":"The numeric formatting is inconsistent, with entries such as '0. 0403' and '0.05700'; standardize the number of decimal places and spacing.","section":"Table 4"},{"comment":"The sentence 'We first compared the performance of several baseline models' has a capitalization issue ('We first' should be 'We first'); please proofread the text throughout.","section":"Section 3.3"},{"comment":"The instruction strategy section only gives a one-sentence example directive and does not specify the full instruction template, hyperparameters, or training details (batch size, sequence length, hardware). Adding these details or explicitly pointing to the released code would improve reproducibility.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The abstract mismatch is the most visible problem and will mislead any reader; it should have been caught before submission. The internal inconsistency between Sections 3.3 and 5 about per-language versus English-only model selection is a substantive methodological ambiguity that undercuts the paper's central explanatory claim. The official leaderboard results are plausible and externally verified, so the paper is worth revising rather than rejecting. I would encourage the authors to either add per-language ablations on the development sets or, if that is not feasible, clearly state that the voting ensemble was chosen on English data and that its cross-lingual advantage is an assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the official leaderboard results are real and clear — 1st in Hindi, 2nd in Russian, 3rd in Portuguese — and the paper is a legitimate SemEval system description. No new method; the value is the empirical comparison of several open LLMs under instruction tuning plus the small single-label vs multi-label analysis. The method is standard, but the external test set gives the numbers independent credibility, and code is linked.\n\nWhat works: the data statistics are useful, the model comparison is plainly laid out, and the paper doesn't oversell the method itself. The single-label experiment, showing only a 2.9% drop when the task is treated as single-label, is a nice data point. For a shared-task paper, the level of transparency is adequate.\n\nSoft spots, in order of seriousness:\n\nFirst, the arXiv metadata abstract is for SemEval Task 7 (claim retrieval), not Task 10. The PDF's own abstract is correct, but anyone landing on the arXiv page gets the wrong summary. That's an editorial screw-up that will actively mislead, and it needs to be fixed before anything else.\n\nSecond, the cross-lingual adaptation is inconsistently described. Section 1 says fine-tune in English then apply to other languages; Section 5 says development data were added to training and the top-performing model was selected; Section 3.3 says top-3 models are selected 'for each language', but the evaluation in Table 3 is English-only. The paper never says what actually changed for Bulgarian, Hindi, Portuguese, and Russian. This is the main substantive weakness: the explanation of the non-English rankings depends on a pipeline that is never pinned down.\n\nThe stress-test concern that the voting benefit rests on English-only selection largely holds. The 0.011 exact-match gain over the best single model on English dev is modest, and there is no evidence it transfers to the other languages. The formal rankings stand regardless, but the paper's claim that voting 'significantly enhances' performance is stronger than the evidence supports.\n\nNone of this sinks the paper. The official results are externally verified, and for a shared-task system description the depth is typical. Send it to review for the workshop, but ask the authors to fix the abstract and to clarify the per-language pipeline. A clean description of what was actually done for each language would also help reproducibility.","headline":"Externally verified shared-task results with a fixable abstract error and an underspecified cross-lingual model-selection story.","tokens_in":5789,"tokens_out":3806,"would_cite":true,"duration_ms":43393,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that an ensemble of instruction-tuned large language models, combined by hard voting, achieves 1st place in Hindi, 2nd in Russian, 3rd in Portuguese, 6th in Bulgarian, and 7th in English on the official test set of…","keywords":["entity framing","multi-label classification","instruction tuning","hard voting","ensemble learning","multilingual","SemEval-2025","large language models"],"falsifier":"Evaluate each candidate single model on the development sets of all five languages and check whether the top-3 by English score match the top-3 by each language's own score; if they differ substantially, the English-selected ensemble is suboptimal for those languages.","tokens_in":4869,"feed_emoji":"🗳️","tokens_out":4765,"duration_ms":47549,"temperature":0.7,"pith_summary":"The paper describes a system for multilingual, multi-label classification of news entity framing, where named entities in news articles are assigned fine-grained roles such as protagonist, antagonist, or innocent. The authors instruction-tune several open large language models on a small English dataset, select the three best on English development data, and combine their predictions with hard voting. In the official evaluation, the ensemble ranks first in Hindi, second in Russian, third in Portuguese, sixth in Bulgarian, and seventh in English, outperforming the official baseline substantially in every language. The central claim is that this simple recipe — English-only model selection plus majority voting — transfers surprisingly well across languages, with residual differences largely tied to training-data size.","feed_headline":"Majority-vote LLMs rank 1st in Hindi, 2nd in Russian","feed_subtitle":"Five-language entity-framing results beat the baseline in every track using instruction-tuned open models.","key_machinery":"The central mechanism is a hard voting ensemble over three instruction-tuned LLMs: Phi-3-small-128k-instruct, Phi-3-medium-128k-instruct, and Phi-4. The models are fine-tuned with a directive instructing the model to assign one or more fine-grained roles to a named entity within an article; the top-3 are chosen by performance on English development data, and their output labels are combined by majority vote, with the most frequent class chosen as the final prediction.","core_discovery":"On the official test set, the authors' instruction-tuned LLM ensemble with hard voting achieves Exact Match Ratio scores of 0.4684 in Hindi (rank 1), 0.5140 in Russian (rank 2), 0.4579 in Portuguese (rank 3), 0.3871 in Bulgarian (rank 6), and 0.3277 in English (rank 7), compared with baselines around 0.04–0.06. On English development data, the voting ensemble reaches 0.4725, improving by 1.1% over the best single model, Phi-4 (0.4615). The paper establishes that combining Phi-3-small, Phi-3-medium, and Phi-4 through majority voting yields consistent though modest gains, and that instruction tuning with a one-to-many directive is the main driver of the large gap over the baseline.","pith_inferences":["A natural extension not explored in the paper is per-language model selection on each language's own development set; if model rankings differ across languages, the English-selected ensemble may be suboptimal for non-English tracks.","The modest size of the single-label drop suggests that label-imbalance handling, such as threshold tuning or asymmetric loss, could squeeze additional gains without changing the model architecture.","The strong correlation between training-data size and final rank hints that data augmentation or cross-lingual transfer from Hindi and Russian could improve English and Bulgarian, which the paper lists as future work."],"forward_implications":["Instruction tuning of open LLMs provides a large improvement over a small DeBERTa baseline for entity framing across multiple languages.","Majority voting over the top-3 models yields consistent but modest gains over the best single model, limited by prediction similarity and task ceiling.","Adding development data to the training set and selecting the top-performing model by evaluation contributes to final test performance.","The gap between high-resource and low-resource languages in final performance appears largely attributable to training-data scale.","The single-label experiment shows only a 2.86% drop, indicating the dataset is heavily single-labeled yet still supports a multi-label formulation."],"supporting_citations":[{"why":"Defines SemEval-2025 Task 10 and subtask 1, providing the task setup, data, and the baseline to beat.","marker":"Piskorski et al., 2025"},{"why":"Defines the fine-grained entity framing label system used as the classification target.","marker":"Mahmoud et al., 2025"},{"why":"Provides Phi-3-small and Phi-3-medium, two of the three ensemble members.","marker":"Abdin et al., 2024a"},{"why":"Provides Phi-4, the best single model and the third ensemble member.","marker":"Abdin et al., 2024b"},{"why":"Provides Qwen2 and Qwen2.5 models used in the model comparison phase.","marker":"Yang et al., 2024"},{"why":"Supplies the hard-voting ensemble technique used to combine the top-3 model predictions.","marker":"Jabbar, 2024"},{"why":"Informs the instruction strategy and the experimental design for instruction tuning.","marker":"Wang et al., 2024b"}],"fun_headline_variants":["Majority-vote LLMs take Hindi crown, Russian silver","Open LLM ensemble ranks 1st in Hindi, 2nd in Russian","Instruction-tuned LLMs beat baselines in multi-language framing","Voting ensemble of LLMs wins top spots in Hindi and Russian","LLM hard voting excels across five languages, topping Hindi"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The top-3 models are selected on English development data alone, and the same ensemble is applied to all languages; if the ranking of models differs across languages, the chosen ensemble may not be the best for the other languages.","fun_headline_variants_meta":{"raw":{"variants":["Majority-vote LLMs take Hindi crown, Russian silver","Open LLM ensemble ranks 1st in Hindi, 2nd in Russian","Instruction-tuned LLMs beat baselines in multi-language framing","Voting ensemble of LLMs wins top spots in Hindi and Russian","LLM hard voting excels across five languages, topping Hindi"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000616,"raw_usage":{"total_tokens":2824,"prompt_tokens":869,"completion_tokens":1955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1865}},"tokens_in":485,"tokens_out":1955,"duration_ms":17689,"temperature":1.0,"reasoning_tokens":1865,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:26:28.043882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate each candidate single model on the development sets of all five languages and check whether the top-3 by English score match the top-3 by each language's own score; if they differ substantially, the English-selected ensemble is suboptimal for those languages.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines SemEval-2025 Task 10 and subtask 1, providing the task setup, data, and the baseline to beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the hard-voting ensemble technique used to combine the top-3 model predictions."}],"review_version":1}