LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles

Egil R{\o}nningstad; Gaurav Negi

REVIEW 2 major objections 6 minor 12 references

LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Entity-to-entity context segments let a fine-tuned masked model beat zero-shot GPT-4o and LoRA-fine-tuned LLMs on narrative role classification.

desk verdict The ent2ent context recipe is genuinely useful and honestly reported, but the abstract's 'outperforms SFT' claim is stronger than the experiments support. read the letter →

arxiv 2506.05976 v1 pith:XVXZOO6R submitted 2025-06-06 cs.CL

Egil R{o}nningstad , Gaurav Negi This is my paper

classification cs.CL

keywords entityframingnarrativerolescontextoptimizationXLM-RoBERTa-largesupervisedfine-tuningLoRAmultilingualclassificationmaskedlanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At issue is whether a model with a 512-token context window can classify the narrative role of an entity in news articles that are much longer than 512 tokens. The paper claims it can, if the input is pre-cropped with a simple rule: take the sentence where the entity appears plus all following sentences up to the next entity mention, and prefix it with the entity's name. Fine-tuned XLM-RoBERTa-large using these 'entity-to-entity' segments reaches 47.75 micro-F1 on the development split, above zero-shot GPT-4o (41.78), LoRA-tuned Llama-3.1-8B (31.78), and LoRA-tuned Mistral-7B (29.52). The paper concludes that context selection rather than model scale is what lets a small multilingual masked model match or exceed supervised fine-tuning of larger generative models.

What carries the argument

The entity-to-entity (ent2ent) context segment. For each annotated entity, the input is the sentence containing the entity plus every later sentence until a new entity appears, prefixed by 'Regarding <entity>:\n'. This gives the model a focused local window with the target entity named, and the ablation without the prefix (micro-F1 drops from 47.75 to 30.11) shows the prefix itself is load-bearing.

What would settle it

The decisive check is to re-run the Llama/Mistral supervised fine-tuning with one prompt per entity (optionally with the same ent2ent context) and see whether either model's dev micro-F1 exceeds 47.75; if so, the paper's central comparison would not survive. A second check: replace the ent2ent boundary with a randomly chosen sentence boundary; if the random crop keeps most of the 47.75 score, the specific boundary rule is not what drives the result.

Watch

Extended reading notes

Core claim

On the SemEval-2025 Task 10 subtask 1 data, the paper's central discovery is that entity-oriented context extraction is the decisive design choice. Fine-tuning XLM-RoBERTa-large on ent2ent segments (the entity sentence plus all later sentences until the next entity) prefixed with 'Regarding <entity>:\n' gives 47.75 micro-F1 on the dev set, the best of every approach the paper compares: full-document XLM-R (38.96), GPT-4o-extracted spans (43.14), single sentence (46.06), paragraph (40.79), a two-stage main-role-first pipeline (44.51), zero-shot ChatGPT-4o (41.78), LoRA-supervised Llama-3.1-8B (31.78), and LoRA-supervised Mistral-7B (29.52). The prefix is not decorative: dropping it lowers micro-F1 to 30.11. The authors therefore claim that a rule-based context-selection heuristic brings a small multilingual masked LM to parity with, or better than, supervised fine-tuning of 7-8B generative models for this task.

Load-bearing premise

The comparison rests on the assumption that the supervised fine-tuning recipe used for Llama and Mistral—one prompt per whole document asking for all entities at once, with LoRA on a causal LM objective—is a fair representative of SFT; the authors themselves note that one-entity-per-prompt or context-optimized SFT might do better but was not tested due to resource limits.

Editorial extensions

If this is right

Fine-tuned masked language models remain competitive with much larger generative models on multilingual entity classification when the input is cropped to an entity-focused span.
A simple deterministic cropping rule can outperform both full-document input and spans extracted by GPT-4o for this task.
Merging all languages for fine-tuning helps every language, including low-resource ones, compared with training on the target language alone.
Removing the entity prefix drops micro-F1 from 47.75 to 30.11, so the identity signal is not optional in this architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the entity prefix likely works as an attention anchor, so varying its wording would separate the contribution of identity signaling from the contribution of span selection.
A second extension: the ent2ent cropping rule should transfer to other multi-entity long-document classification tasks, such as event-role labeling or stance attribution, where a 512-token model must focus on local context.
A third extension: because the paper did not run SFT with one entity per prompt, a fair re-run of Llama/Mistral with ent2ent context is the immediate test that could overturn or confirm the headline comparison.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a useful, plainly written system description with one genuinely reusable idea — the entity-to-entity context extraction heuristic — and one overstated headline claim. The stress-test note lands: the SFT comparison is set up at a disadvantage, and the paper itself says so in Section 4.3.

What's actually new: the ent2ent rule (entity mention plus following sentences up to the next entity, prefixed with the mention) is simple, parameter-free, and reproducible. The ablations in Table 2 do real work: the prefix matters enormously (47.75 vs 30.11 without it), full text hurts (38.96), and the cheap heuristic beats a GPT-4o extraction baseline (43.14) at zero API cost. That's a practical result for low-resource multilingual classification, and the multilingual-vs-monolingual comparison (Table 3, Figure 3) is a nice bonus. I also credit the authors for flagging in Section 4.3 that their SFT setup — one joint prompt per document, all entities at once, no context optimization — could be improved. That is the main weakness staring out of the paper.

The soft spots, in proportion: the abstract's comparative claim rests on an asymmetric baseline. XLM-R sees one entity at a time with distilled context; Llama and Mistral are asked to solve a harder joint decoding problem from full documents. A reasonably tuned SFT baseline with per-entity prompts or the same context optimization might close or flip the gap. So the honest summary is: context selection makes a small MLM surprisingly strong, not that it beats a fair SFT setup. Second, all numbers are single runs without variance, and the winning heuristic was chosen on the dev split that is also used for reporting; that's a mild selection effect, not a fatal flaw, but the 47.75 should be read as a point estimate. Third, the official test-set results appear only as rankings, not scores, which weakens the paper as a standalone artifact.

Who gets value: people working on entity-level classification in low-resource or multilingual settings, and anyone building shared-task baselines who wants a cheap, well-ablated context-selection recipe. It's not a conceptual breakthrough, but it doesn't need to be.

Recommendation: yes, send it to peer review — it's a legitimate empirical comparison with an honest limitations section. A referee should ask for reworded claims, test-set numbers, and ideally variance estimates. The central context-selection finding holds; the SFT comparison needs a softer statement.

Referee Report

2 major / 6 minor

Summary. This paper describes the LTG submission to SemEval-2025 Task 10, subtask 1, on entity framing in multilingual news articles. The authors compare several rule-based context-extraction strategies for fine-tuning XLM-RoBERTa-large, including single sentence, paragraph, full text, entity-to-entity (ent2ent) segments, and GPT-4o-extracted spans, with an entity-mention prefix prepended to the selected segment. They also report zero-shot ChatGPT-4o classification and supervised fine-tuning (SFT) of Llama-3.1-8B and Mistral-7B using 4-bit LoRA. On the dev set, the ent2ent-prefixed XLM-R obtains the best overall micro-F1 (47.75), outperforming the other XLM-R context variants and the SFT baselines (31.78 and 29.52). The paper argues that simple entity-oriented context optimization enables masked language models with limited context windows to compete with larger generative models.

Significance. The empirical demonstration that simple entity-oriented segment extraction improves performance for a masked language model is a useful practical contribution to shared-task system development. The ablation across context-extraction strategies (Table 2) is informative and supports the conclusion that context selection matters for XLM-R. However, the headline comparative claim against SFT is weakened by the acknowledged asymmetry in experimental setup: the XLM-R system classifies one entity at a time with a distilled context segment, while the SFT baselines use one joint prompt per document with no context optimization. The contribution is best read as a system description rather than a general finding that small MLMs outperform SFT of larger LLMs. Reproducibility is further limited by the lack of variance estimates and by dev-set-based method selection, but the core finding about context selection is likely robust.

major comments (2)

[§3.3 and §4.3, Table 2] The comparison with SFT is not controlled. The XLM-R system classifies each entity separately with an ent2ent context segment and an entity prefix, whereas the Llama and Mistral SFT baselines are given one joint prompt per document and must output roles for all entities at once, without context optimization. The paper explicitly acknowledges in Section 4.3 that per-entity prompting or applying context optimization to SFT "might prove beneficial" and was not tested due to resource limits. The gap between 47.75 and 31.78/29.52 therefore conflates the benefit of per-entity decomposition and context selection with the choice of base model. The abstract's claim that the approach "outperforms Supervised Fine-Tuning with larger generative language models" is not supported by the presented evidence; it should be qualified or supported by a more competitive SFT baseline.
[Table 2 and §4.1] The ranking of context-extraction strategies is based on single runs with no error bars or significance tests, and the best strategy (ent2ent) was selected by inspecting the same dev set used for the final reported numbers. With per-language dev sets as small as 30 (BG) and 91 (EN), micro-F1 differences on the order of 1–2 points (e.g., ent2ent 47.75 vs. sentence 46.06) are plausibly within noise. This selection effect inflates the reported performance of ent2ent and makes the "best overall" claim fragile. Reporting multiple seeds or confidence intervals, and ideally holding out a split for method selection, would strengthen the central claim.

minor comments (6)

[Section 1.2] "has proven to to be" is a typo; it should read "has proven to be".
[Abstract] "a simple entity-oriented heuristics" should be "a simple entity-oriented heuristic".
[Section 2.1(d)] The definition of ent2ent should clarify what happens when no subsequent entity appears before the end of the document, and whether the sentence containing the entity is always included even if it also introduces a new entity.
[Table 3] The column headers "language all in-lang samples" are unclear, and the "all" row appears to be missing the in-lang value. Please reformat the table for readability.
[Figure 1] The caption refers to lines for Llama and Mistral, but the figure description does not include a legend or markers to distinguish the two models; please add them.
[Section 3.1] Stating that no hyperparameters were changed from the Transformers default is insufficient for reproducibility; please report the learning rate, batch size, and number of training steps or epochs used for the XLM-R runs.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the paper is an empirical comparison, and its main weaknesses are experimental-fairness caveats rather than circular reasoning.

full rationale

The paper contains no derivation chain that reduces to its own inputs. The central claim is an empirical comparison: fine-tuned XLM-RoBERTa-large with the rule-based ent2ent context heuristic achieves 47.75 dev micro-F1, compared with 31.78 for Llama-3.1-8B SFT, 29.52 for Mistral-7B SFT, and 41.78 for zero-shot ChatGPT-4o (Table 2). The ent2ent heuristic is defined independently of the outcome (Section 2.1d: provide the sentence where the entity is mentioned and all subsequent sentences until a new entity occurs), and the XLM-R setup uses standard Transformers defaults, so no parameter is fitted to the predicted quantity and no result is equivalent to its input by construction. The abstract's comparative claim versus SFT is weakened by an implementation asymmetry: the SFT models receive one joint prompt per document, requiring all entities at once, while XLM-R receives per-entity context segments. Section 4.3 explicitly concedes that requesting one entity per prompt or applying context optimization to SFT 'might prove beneficial' but was beyond resource allocations. This is a legitimate external-validity limitation, not circularity. The only self-citation (Rønningstad, 2023, Section 2.2) supports a general multilingual fine-tuning tradeoff and is not load-bearing for the main result. Selecting the best context heuristic on the dev split is a selection effect, not circular reasoning, and the paper also reports official test-set ranks. Overall, the claims are self-contained empirical findings with independent baselines and no circular reduction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper rests on standard empirical assumptions: the dev split represents the test distribution, XLM-R's 512-token window is sufficient when segments are extracted, the entity indices in the annotations are correct, and the SFT prompt design is a fair baseline. The authors themselves flag the last assumption as questionable in Section 4.3.

assumptions (4)

domain assumption The dev split is representative of the test distribution for all five languages.
The paper reports dev-set results as evidence for the method's effectiveness, implying that dev performance generalizes to test. The authors only mention official test-set rankings without reporting scores, so this is an unverified assumption.
domain assumption XLM-R's 512-token context window is sufficient for the extracted segments.
The ent2ent extraction is designed to fit within XLM-R's window; if a relevant framing is spread beyond the next entity mention, the method would miss it.
domain assumption The provided entity indices and annotations are correct.
The context extraction relies on entity spans and sentence boundaries from the task annotations; incorrect indices would corrupt the extracted segments.
domain assumption The SFT setup for Llama and Mistral is a fair and representative implementation of supervised fine-tuning.
The paper's abstract claims outperformance, but Section 4.3 admits the prompt design (all entities in one prompt) may be suboptimal, and alternatives were not tested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles." pith.science (2026). https://pith.science/paper/XVXZOO6R

@misc{pith2026250605976,
  author       = {Pith},
  title        = {Pith review of: LTG at SemEval-2025 Task 10: Optimizing Context for Classification of Narrative Roles},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVXZOO6R}},
  note         = {Machine review of arXiv:2506.05976}
}

read the original abstract

Our contribution to the SemEval 2025 shared task 10, subtask 1 on entity framing, tackles the challenge of providing the necessary segments from longer documents as context for classification with a masked language model. We show that a simple entity-oriented heuristics for context selection can enable text classification using models with limited context window. Our context selection approach and the XLM-RoBERTa language model is on par with, or outperforms, Supervised Fine-Tuning with larger generative language models.

Figures

Figures reproduced from arXiv: 2506.05976 by the authors.

**Figure 2.** Model performance as a function of inlanguage training data need for more than a thousand in-language, indomain training samples when fine-tuning XLMR, is in line with previous experience. But again, XLM-R performs best of our approaches for Russian as well, with the fewest training samples. We can assume that languages’ presence in the original model pretraining also contributes much to the quality of the resu… view at source ↗

**Figure 3.** Benefits from training on all languages. For each language, the results improve substantially when [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Hongjie Cai, Heqing Ma, Jianfei Yu, and Rui Xia. 2024. https://doi.org/10.18653/v1/2024.acl-long.657 A joint coreference-aware approach to document-level target sentiment analysis . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12149--12160, Bangkok, Thailand. Association for Comp...

work page doi:10.18653/v1/2024.acl-long.657 2024
[4]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

doi:10.18653/v1/2020.acl-main.747 2020
[5]

Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane Luke. 2024. Prompt compression with context-aware sentence encoding for fast and improved llm inference. arXiv preprint arXiv:2409.01227

arXiv 2024
[6]

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. https://doi.org/10.18653/v1/2023.findings-acl.779 Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 12284--12314, Toronto, Canada. Association for Computation...

doi:10.18653/v1/2023.findings-acl.779 2023
[7]

Jakub Piskorski, Tarek Mahmoud, Nikolaos Nikolaidis, Ricardo Campos, Jorge Alípio, Dimitar Dimitrov, Purificação Silvano, Roman Yangarber, Shivam Sharma, Tanmoy Chakraborty, Nuno Ricardo Guimarães, Elisa Sartori, Nicolas Stefanovitch, Zhuohan Xie, Preslav Nakov, and Giovanni Da San Martino. 2025. SemEval -2025 task 10: Multilingual characterization and ex...

work page 2025
[8]

Egil R nningstad. 2023. https://doi.org/10.18653/v1/2023.semeval-1.144 UIO at S em E val-2023 task 12: Multilingual fine-tuning for sentiment classification in low-resource languages . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1054--1060, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.semeval-1.144 2023

Show all 12 references

[9]

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Sch \"a rli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210--31227. PMLR

2023
[10]

Nicolas Stefanovitch, Tarek Mahmoud, Nikolaos Nikolaidis, Jorge Alípio, Ricardo Campos, Dimitar Dimitrov, Purificação Silvano, Shivam Sharma, Roman Yangarber, Nuno Guimarães, Elisa Sartori, Ana Filipa Pacheco, Cecília Ortiz, Cláudia Couto, Glória Reis de Oliveira, Ari Gonçalve...

2025
[11]

Jinlin Wang, Suyuchen Wang, Ziwen Xia, Sirui Hong, Yun Zhu, Bang Liu, and Chenglin Wu. 2024. Fact: Examining the effectiveness of iterative context rewriting for multi-fact retrieval. arXiv preprint arXiv:2410.21012

2024 arXiv
[12]

Zijun Wu, Bingyuan Liu, Ran Yan, Lei Chen, and Thomas Delteil. 2024. Reducing distraction in long-context language models by focused learning. arXiv preprint arXiv:2411.05928

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Hongjie Cai, Heqing Ma, Jianfei Yu, and Rui Xia. 2024. https://doi.org/10.18653/v1/2024.acl-long.657 A joint coreference-aware approach to document-level target sentiment analysis . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12149--12160, Bangkok, Thailand. Association for Comp...

work page doi:10.18653/v1/2024.acl-long.657 2024

[4] [4]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...

doi:10.18653/v1/2020.acl-main.747 2020

[5] [5]

Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane Luke. 2024. Prompt compression with context-aware sentence encoding for fast and improved llm inference. arXiv preprint arXiv:2409.01227

arXiv 2024

[6] [6]

Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. 2023. https://doi.org/10.18653/v1/2023.findings-acl.779 Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation . In Findings of the Association for Computational Linguistics: ACL 2023, pages 12284--12314, Toronto, Canada. Association for Computation...

doi:10.18653/v1/2023.findings-acl.779 2023

[7] [7]

Jakub Piskorski, Tarek Mahmoud, Nikolaos Nikolaidis, Ricardo Campos, Jorge Alípio, Dimitar Dimitrov, Purificação Silvano, Roman Yangarber, Shivam Sharma, Tanmoy Chakraborty, Nuno Ricardo Guimarães, Elisa Sartori, Nicolas Stefanovitch, Zhuohan Xie, Preslav Nakov, and Giovanni Da San Martino. 2025. SemEval -2025 task 10: Multilingual characterization and ex...

work page 2025

[8] [8]

Egil R nningstad. 2023. https://doi.org/10.18653/v1/2023.semeval-1.144 UIO at S em E val-2023 task 12: Multilingual fine-tuning for sentiment classification in low-resource languages . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1054--1060, Toronto, Canada. Association for Computational Linguistics

work page doi:10.18653/v1/2023.semeval-1.144 2023