REVIEW 3 major objections 5 minor 19 references
FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A frozen multilingual sentence encoder feeding a deep regression head can rank word-use pairs by annotator disagreement, and the resulting system finished third among seven teams in CoMeDi Subtask 2.
desk verdict A modest shared-task system report with real empirical scores but an internally inconsistent 'ranked 3rd' claim that needs fixing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the concatenated embedding pair $X=[E(C_1),E(C_2)]$, where each $E(C)$ is the 768-dimensional vector from the frozen paraphrase-xlm-r-multilingual-v1 sentence encoder built on XLM-RoBERTa. The regression head maps this vector through four fully connected layers with widths 512, 256, 128, and 64, each followed by batch normalization, ReLU, and dropout at rate 0.3, then to a single output trained with mean squared error. This mechanism converts a pair of contexts into a scalar disagreement score without any task-specific tuning of the underlying encoder, so the ranking signal must come from the pretrained embeddings plus the learned head.
What would settle it
Replacing the deep MLP with a plain linear regression on the same 1,536-dimensional concatenated embeddings would settle whether the architecture contributes anything: if linear regression matches or beats the reported Spearman values on each language, the ranking signal lives in the embeddings, not the deep head. Conversely, fine-tuning only the sentence encoder on the mean-disagreement objective and observing whether the Spanish (0.012) and Chinese (0.018) correlations recover would test whether the frozen-embedding choice is the main bottleneck.
Extended reading notes
Core claim
The paper's central claim, stated on its own terms, is that disagreement ranking in a multilingual word-in-context setting reduces to a regression problem on concatenated contextual embeddings. For each word-use pair, the system takes the 768-dimensional sentence embedding of each context from paraphrase-xlm-r-multilingual-v1, concatenates them into a 1,536-dimensional vector, and trains a four-hidden-layer feedforward network with batch normalization and dropout to predict the mean pairwise absolute difference of the annotators' ordinal judgments. The paper reports that this explicit modeling of disagreement, rather than aggregation toward a gold label, produced competitive Spearman correlations and ranked third of seven teams in the evaluation phase. The same setup is offered as evidence that robust multilingual embeddings and a regularized deep regression head are sufficient ingredients for this task.
Load-bearing premise
The load-bearing premise is that the frozen sentence encoder's embeddings carry enough signal about a word-use pair to predict how much annotators will disagree; if that premise fails, the whole pipeline fails regardless of the regression head.
Editorial extensions
If this is right
- Other teams can reproduce a competitive multilingual disagreement ranker by combining a publicly available sentence encoder with a small regularized regression head, without fine-tuning the encoder.
- Because the regression target is the mean pairwise absolute judgment difference, the same architecture could be retrained on other annotation tasks where disagreement, not consensus, is the quantity of interest.
- The per-language evaluation results imply that frozen multilingual embeddings transfer unevenly: languages with longer or typologically different contexts, such as Spanish, may need language-specific adaptation to avoid near-zero correlation.
- The system's third-place ranking in a hidden evaluation phase suggests that simpler embedding-plus-regression baselines can serve as a strong reference point for future disagreement-ranking systems.
Reading between the lines
- A natural test the paper does not run is to compare against a linear regression on the same concatenated embeddings; if the linear model matches the deep head's Spearman values, the reported success would be attributable to the embeddings rather than the architecture.
- The near-zero Spanish and Chinese scores suggest that the sentence-level pooling may discard the target-word information that disagreement depends on; a token-level pooling around the target word indices could be investigated as a fix.
- One could extend the evaluation to per-language Spearman values across multiple training seeds, since the paper reports no variance and it is unclear whether the 0.012 and 0.018 scores are stable or noise.
- Because the encoder is frozen, the system leaves on the table any gains from task-adaptive training; an immediate extension is to fine-tune a lightweight adapter or the upper transformer layers on the mean-disagreement objective.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes FuocChuVIP123's submission to the CoMeDi Shared Task, Subtask 2 (DisWiC), which asks systems to rank word-in-context pairs by the mean pairwise disagreement of annotator judgments across seven languages. The system concatenates 768-dimensional sentence embeddings from the frozen paraphrase-xlm-r-multilingual-v1 model and trains a four-hidden-layer MLP with batch normalization, dropout, and an MSE loss to regress the mean disagreement score. Performance is measured by Spearman correlation against the official labels. The paper reports evaluation-phase scores in Table 3 and claims that the system achieved competitive performance, ranking 3rd among 7 teams.
Significance. If the reported evaluation-phase scores are accurate, the paper documents a simple and transparent baseline: frozen off-the-shelf multilingual sentence embeddings plus a small regression head. The main value is empirical: this approach yields moderate correlations for German and Swedish but near-zero correlations for Spanish and Chinese, which is a useful data point for shared-task participants. The architecture description is clear enough to reproduce. However, the interpretation is overstated: the average rank shown in the paper's own table is 4th, not 3rd, and the 'competitive performance' language is not supported by the near-random results on two languages. With a corrected and more carefully framed presentation, the paper could be an acceptable system-description contribution, but the current central claim needs to be reconciled with the data.
major comments (3)
- [Abstract; §6; Table 3] The headline claim that the system 'ranked 3rd out of 7 teams' in the evaluation phase is not supported by Table 3. The table lists FuocChuVIP123 with an average Spearman correlation of 0.124 and a parenthetical rank of (4), and per-language ranks of (4) for ZH, (7) for EN, (3) for DE, (4) for NO, (6) for RU, (7) for ES, and (3) for SV. No aggregation rule, tie-breaking rule, or official leaderboard excerpt is provided that would convert these values into an overall rank of 3. Please reconcile the stated rank with Table 3 or revise the Abstract and Conclusion accordingly.
- [§5] The training and evaluation description is ambiguous in a way that affects reproducibility and the validity of model selection. The text says 'We merged the necessary information from training and development sets to construct the input for our model' and later 'Our model was evaluated on the development set,' while §4.4 describes an 80/20 training/validation split. It is not clear whether the development set was used for training, for model selection, or only for final evaluation, or what exactly was merged. Please specify the exact data flow—which splits were used for fitting, early stopping, and reporting—and state whether any development-set labels influenced the final model.
- [§6; §8] The explanation of the system's weak performance is inconsistent with Table 3. Section 8 says the system 'struggled with Latin-based languages like Spanish,' and Section 6 refers to challenges with 'Latin languagues,' but the two worst per-language correlations in Table 3 are Chinese (0.018) and Spanish (0.012), while English (0.078) also ranks last among the seven systems listed. The claim about language families is not supported by an analysis; either provide language-level diagnostics or withdraw the explanatory claim.
minor comments (5)
- [Tables 1 and 2] The sample counts in Table 1 are printed as decimals (20.46, 10.83, 6.04, etc.), whereas Table 2 lists integers (871, 1,932, 1,269) and §3.1 says counts range from 1,222 to 24,891; please clarify the units or formatting (e.g., thousands) and make the two tables consistent.
- [§6] There is a typo: 'languagues' should be 'languages.'
- [§2] The reference to (Mostafazadeh Davani et al., 2022) is duplicated in the same sentence; please cite the relevant work once and distinguish it from any other cited work by the same authors.
- [§4.3] The description of XLM-RoBERTa as ending with a sigmoid activation and cross-entropy loss describes a classification head that is not part of the regression system used in this paper; consider removing this passage or explicitly labeling it as background.
- [Table 3] The column header 'A VG' appears to be a typo for 'Avg,' and the average column should be defined (e.g., whether it is the mean of the per-language Spearman values).
Circularity Check
No circularity: the paper reports a standard supervised system evaluation on an external shared-task benchmark, with predictions measured against held-out labels.
full rationale
The paper is a shared-task system description. Its derivation chain is straightforward: frozen off-the-shelf sentence embeddings (paraphrase-xlm-r-multilingual-v1) are concatenated; a feedforward regression network is trained with MSE loss on the provided training split to predict mean pairwise disagreement; and performance is reported as Spearman correlation against held-out evaluation labels in Table 3. Every model parameter is fitted to the training target in ordinary supervised fashion, and the reported evaluation scores are computed on data not used for fitting. No equation in the paper defines the predicted score as a fitted value, nor does any definition reduce the target quantity to an input quantity. Citations to Schlechtweg et al. for the task and datasets, Conneau et al. for XLM-R, and Zar for Spearman correlation are external references and are not load-bearing in a way that makes the result circular. The only noteworthy issue is an internal consistency discrepancy between the claim of ranking 3rd out of 7 teams (Sections 6 and 7) and Table 3, where the system's average Spearman value of 0.124 is labelled with rank (4); that is a factual or reporting concern, not circularity, and it does not affect the circularity score.
Assumptions & free parameters
free parameters (5)
- dropout probability p =
0.3
- initial learning rate =
1e-4
- hidden layer sizes =
[512, 256, 128, 64]
- batch size =
32
- training epochs =
17
assumptions (3)
- domain assumption The concatenation [E(C1), E(C2)] contains sufficient signal to predict mean pairwise disagreement.
- domain assumption A fixed, pre-trained multilingual sentence embedding generalizes across the seven languages without encoder fine-tuning.
- domain assumption Mean pairwise absolute difference of annotator judgments is the correct target for disagreement ranking.
Cite this review
Pith. "Pith review of FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression." pith.science (2026). https://pith.science/paper/ZEGYICJC
@misc{pith2026250112336,
author = {Pith},
title = {Pith review of: FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZEGYICJC}},
note = {Machine review of arXiv:2501.12336}
}
read the original abstract
This paper presents results of our system for CoMeDi Shared Task, focusing on Subtask 2: Disagreement Ranking. Our system leverages sentence embeddings generated by the paraphrase-xlm-r-multilingual-v1 model, combined with a deep neural regression model incorporating batch normalization and dropout for improved generalization. By predicting the mean of pairwise judgment differences between annotators, our method explicitly targets disagreement ranking, diverging from traditional "gold label" aggregation approaches. We optimized our system with a customized architecture and training procedure, achieving competitive performance in Spearman correlation against mean disagreement labels. Our results highlight the importance of robust embeddings, effective model architecture, and careful handling of judgment differences for ranking disagreement in multilingual contexts. These findings provide insights into the use of contextualized representations for ordinal judgment tasks and open avenues for further refinement of disagreement prediction models.
Figures
Reference graph
Works this paper leans on
-
[1]
Ron Artstein and Massimo Poesio. 2008. https://doi.org/10.1162/coli.07-034-R2 Survey article: Inter-coder agreement for computational linguistics . Computational Linguistics, 34(4):555--596
-
[2]
Valerio Basile, Federico Cabitza, Andrea Campagner, and Michael Fell. 2021. https://arxiv.org/abs/2109.04270 Toward a perspectivist turn in ground truthing for predictive computing . CoRR, abs/2109.04270
arXiv 2021
-
[3]
Jing Chen, Emmanuele Chersoni, Dominik Schlechtweg, Jelena Prokic, and Chu-Ren Huang. 2023. https://aclanthology.org/2023.lchange-1.10/ ChiWUG : A graph-based evaluation dataset for C hinese lexical semantic change detection . In Proceedings of the 4th International Workshop on Computational Approaches to Historical Language Change, Singapore. Association...
work page 2023
-
[4]
Xiangyi Chen, Steven Z. Wu, and Mingyi Hong. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/9ecff5455677b38d19f49ce658ef0608-Paper.pdf Understanding gradient clipping in private sgd: A geometric perspective . In Advances in Neural Information Processing Systems, volume 33, pages 13773--13782. Curran Associates, Inc
work page 2020
-
[5]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Comp...
-
[6]
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. https://aclanthology.org/N13-1132 Learning whom to trust with MACE . In Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1120--1130, Atlanta, Georgia. Association for Computationa...
work page 2013
-
[7]
Klaus Krippendorff. 2018. Content Analysis: An Introduction to Its Methodology, 4 edition. SAGE Publications
work page 2018
-
[8]
Sinan Kurtyigit, Maike Park, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. https://aclanthology.org/2021.acl-long.543/ Lexical Semantic Change Discovery . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lon...
work page 2021
Show all 19 references
-
[9]
Andrey Kutuzov, Samia Touileb, Petter M hlum, Tita Enstad, and Alexandra Wittemann. 2022. https://aclanthology.org/2022.lrec-1.274 N or D ia C hange: Diachronic semantic change dataset for N orwegian . In Proceedings of the Thirteenth Language Resources and Evaluation Conferen...
2022
-
[10]
Aida Mostafazadeh Davani, Mark D \' az, and Vinodkumar Prabhakaran. 2022. https://doi.org/10.1162/tacl_a_00449 Dealing with disagreements: Looking beyond the majority vote in subjective annotations . Transactions of the Association for Computational Linguistics, 10:92--110
2022 doi
-
[11]
Julia Rodina and Andrey Kutuzov. 2020. https://doi.org/10.18653/v1/2020.coling-main.90 R u S em S hift: a dataset of historical lexical semantic change in R ussian . In Proceedings of the 28th International Conference on Computational Linguistics, pages 1037--1047, Barcelona, ...
2020 doi
-
[12]
Dominik Schlechtweg, Pierluigi Cassotti, Bill Noble, David Alfter, Sabine Schulte im Walde, and Nina Tahmasebi. 2024. More DWUGs : Extending and evaluating word usage graph datasets in multiple languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural La...
2024
-
[13]
Dominik Schlechtweg, Tejaswi Choppa, Wei Zhao, and Michael Roth. 2025. The CoMeDi shared task: Median judgment classification & mean disagreement ranking with ordinal word-in-context judgments. In Proceedings of the 1st Workshop on Context and Meaning--Navigating Disagreements...
2025
-
[14]
Dominik Schlechtweg, Sabine Schulte im Walde , and Stefanie Eckmann. 2018. https://www.aclweb.org/anthology/N18-2027/ Diachronic Usage Relatedness (DURel) : A framework for the annotation of lexical semantic change . In Proceedings of the 2018 Conference of the North American ...
2018
-
[15]
Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2021. https://api.semanticscholar.org/CorpusID:245589751 Learning from disagreement: A survey . J. Artif. Intell. Res., 72:1385--1470
2021
-
[16]
Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg
Frank D. Zamora-Reina, Felipe Bravo-Marquez, and Dominik Schlechtweg. 2022. https://aclanthology.org/2022.lchange-1.16/ LSCDiscovery : A shared task on semantic change discovery and detection in S panish . In Proceedings of the 3rd International Workshop on Computational Appro...
2022
-
[17]
Jerrold H. Zar. 2005. https://api.semanticscholar.org/CorpusID:119015625 Spearman rank correlation
2005
-
[18]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[19]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.