REVIEW 4 major objections 5 minor 30 references
USTCCTSU at SemEval-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that adding a whitening transform to XLM-RoBERTa-base sentence embeddings, together with a language-level training-data filter built from unlabeled target-language text, is what produces top rankings in cross-lingual…
desk verdict Competition result is real, but the paper's attribution of the gain to whitening is confounded by the data-filtering step and by fitting whitening parameters to the test set itself. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the whitening transform $\tilde{s}_i = (s_i - \mu) W$, with $\mu$ the mean vector and $W = U\Lambda^{-1/2}$ obtained from the singular value decomposition of the covariance matrix $\Sigma = U\Lambda U^\top$; this maps the sentence-vector cloud to zero mean and identity covariance. It is applied to frozen XLM-RoBERTa-base representations, with a top-$k$ cutoff of 256. The second module is a data-selection rule: using unlabeled target-language text to compute the whitening parameters, the paper computes the Spearman correlation for each candidate training language, keeps languages whose inclusion improves prediction, and drops those that hurt, on the assumption that language influence is mutual.
What would settle it
Estimate the whitening mean and covariance from target-language test-distribution text versus only from source-language text, run the same frozen encoder and filtering rule under both conditions, and compare the final Spearman correlations; if the gap nearly vanishes, the anisotropy fix is not the main driver of the reported leaderboard results.
Extended reading notes
Core claim
The central claim is that the poor cosine-similarity behavior of multilingual sentence encoders is fixable at inference time, without fine-tuning, by whitening the embedding space, and that training-set composition can be improved by deleting languages that hurt rather than help. Concretely, the paper shows that XLM-RoBERTa-base with whitening outperforms the same encoder without it (Indonesian 0.4746 vs 0.4390; Spanish 0.6886 vs 0.5907), and that dropping most training languages in favor of two better-matched languages raises Spanish performance from 0.6375 to 0.6886. The system achieves 2nd place in Spanish and 3rd in Indonesian in Track C of SemEval-2024 Task 1.
Load-bearing premise
The load-bearing premise is that the system may compute the whitening mean and covariance from unlabeled text in the exact target language at test time; remove that access and the method as described cannot be applied.
Editorial extensions
If this is right
- Whitening is a no-fine-tuning post-processing step that can be dropped into frozen multilingual encoders to improve relatedness scoring.
- Filtering the training data by measured language influence lets a system use fewer languages and obtain better results, easing the curse of multilingualism.
- The measured cosine-similarity shift from a narrow concentrated band to a wider distribution indicates that anisotropy, not model capacity alone, was limiting the baseline.
- The approach transfers to target languages without requiring labeled target-language data, needing only unlabeled text from the target language.
- The same whitening-plus-selection recipe can be evaluated on other cross-lingual tasks where embeddings are compared by cosine similarity.
Reading between the lines
- The reported rankings assume the system can see unlabeled text from the exact target language before scoring; in a deployment where no such text exists, the whitening step would need a proxy corpus, and the gains could shrink.
- The data-filtering heuristic is separable from whitening and could be tested as a standalone training-set selection method for other multilingual transfer tasks.
- The fixed top-$k$ of 256 suggests that the choice of how many whitened dimensions to keep may matter as much as the whitening itself, inviting experiments with a learned rank rather than a fixed cutoff.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes the USTCCTSU submission to SemEval-2024 Task 1 Track C, a cross-lingual semantic textual relatedness task. The system freezes XLM-R-base, applies a whitening transformation to the resulting sentence embeddings to reduce anisotropy, and uses a language-filtering method that selects training languages based on the target language's unlabeled text. The authors report Spearman correlations on the Indonesian and Spanish test sets, claiming a 2nd-place result in Spanish and 3rd in Indonesian, and attribute the gains to whitening and data filtering. The paper also includes a cosine-similarity distribution analysis intended to show that whitening makes the embedding space more isotropic.
Significance. The competition results are externally supported: the reported ranks in Track C are verifiable leaderboard outcomes, and the paper provides a clear presentation of the whitening formulas and the system configuration. If the causal claims were established, the work would be a useful case study in applying whitening and training-set selection to cross-lingual semantic relatedness. However, the current evidence does not isolate the effect of whitening from the data-filtering step or from access to the target-language test distribution, and no statistical reliability checks are reported. The significance is therefore conditional on a revision that provides ablations and clarifies the transductive nature of the procedure.
major comments (4)
- [Section 4, Table 1] The attribution of the gain to whitening is confounded. The Spanish value reported for XLM-Rbase-whitening in Table 1 (0.6886) is exactly the value reported in Section 3.4 for the training set with data filtering, so the row labeled 'XLM-Rbase-whitening' includes the data-filtering step. The comparison against the baseline therefore conflates at least three factors: the whitening transform, the use of target-language test data to estimate the transform, and the language filtering. No condition is reported for the model with filtering but without whitening, nor for whitening whose parameters are estimated from source-language or development data. The central claim that whitening itself improves cross-lingual semantic relatedness cannot be evaluated from the presented table.
- [Section 4] The whitening parameters mu and W are computed using the target-language text from Track C, and the same target-language test data are then used to compute the reported Spearman correlations. This is a transductive setup: the representations are fitted to the test distribution on which they are evaluated. Consequently, the reported scores do not demonstrate that the whitening step transfers to new target languages or that the observed gains would persist under a distribution shift. If the competition rules permit the use of unlabeled test sentences, this should be stated explicitly and its implications for the generalization claim should be discussed; otherwise, the experimental protocol does not support the paper's conclusions.
- [Section 3.4] The data-filtering method rests on an unvalidated symmetry assumption: the paper asserts that if including target-language text degrades prediction on a training language, then that training language degrades prediction on the target language. No evidence is provided for this reversibility. Moreover, the filtering decision is made using whitening parameters computed from the target test text, so the selected training languages are chosen specifically for the whitening transform that will be used on that test set. A held-out validation step is needed to show that the filtering procedure improves target-language performance rather than merely selecting languages that work well with a test-set-specific transform.
- [Table 1, Section 5] The reported improvements lack statistical reliability measures. The Indonesian gain from baseline to XLM-Rbase-whitening is 0.4746 versus 0.4700, a difference of 0.0046, and no error bars, repeated runs, or significance tests are reported. Given that the whitening row also includes filtering and test-distribution access, this difference is far too small to support the claim that whitening 'significantly improves' the task. The authors should report variance across runs or at minimum justify that the differences exceed the noise level of the evaluation.
minor comments (5)
- [Section 3.3] The derivation of the whitening transform largely follows Su et al. (2021), but the notation is occasionally inconsistent: for example, equation (1) uses x_i while s_i was introduced as the sentence vector. Please align the notation throughout.
- [Section 4] The paper does not explain what the 'topk parameter of whitening' means or how the value 256 was chosen. Since whitening normally uses the full covariance matrix, the truncation to 256 components and its effect on results should be clarified.
- [Section 3.4] The description of the filtering example for Spanish is confusing: the phrase '1,000 each of all data except Spanish' is followed by '1,000 each of kin and ind', but it is not clear whether the final Spanish training set contains only two languages or whether other languages were considered and excluded. Please specify the exact language sets before and after filtering.
- [Figure 2] Figure 2 does not specify which language or data split the cosine-similarity histograms are computed on, nor whether the same sentence pairs are used in both panels. Adding this information would make the anisotropy claim easier to verify.
- [General] There are numerous typographical and formatting issues, including 'XLM -Rbase' with inconsistent spacing, 'RoBERT' instead of 'RoBERTa', 'are were significantly improved', 'We verifies', and an incomplete reference entry for 'Wang et al.'. A careful proofreading pass is needed.
Circularity Check
No definitional circularity; transductive whitening and confounded ablation are experimental-design concerns, not circularity.
full rationale
The whitening transformation in Section 3.3 is mathematically self-contained: Eqs. (1)-(10) define a standard whitening map from any vector set S, and the claim that it produces an isotropic distribution is true by construction. The empirical claim that whitening improves relatedness is based on Table 1, but that table's XLM-Rbase-whitening row also includes the data-filtering method, and Section 4 states that the whitening mean and covariance are computed from the target-language track C text. This is a real attribution and transductive-bias concern, but it is not circularity under the required standard: the whitening parameters are unsupervised, do not use relatedness labels, and the reported Spearman correlations are not determined by the construction of mu and W. The data-filtering step uses the target test text for whitening when selecting training languages, yet the final evaluation still uses gold labels only for scoring, so the derived metric is not equivalent to the fitted inputs. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation are present. The weaknesses are experimental-design and generalization issues, not a derivation that reduces to its own inputs.
Assumptions & free parameters
free parameters (3)
- topk (whitening component count) =
256
- whitening mean vector and covariance matrix (mu, W) =
Computed from target-language test sentences
- training data size per language =
1,000
assumptions (4)
- domain assumption Whitening (zero mean, unit covariance) improves cosine-similarity based relatedness estimation for XLM-R embeddings.
- ad hoc to paper Language interaction in multilingual representation spaces is symmetric: if target text degrades prediction on a training language, that training language degrades prediction on the target.
- domain assumption The curse of multilingualism applies to this training data and explains the benefit of removing languages.
- domain assumption Unlabeled target-language test text may be used to estimate whitening parameters and to drive filtering decisions.
Cite this review
Pith. "Pith review of USTCCTSU at SemEval-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task." pith.science (2026). https://pith.science/paper/LETX3MUK
@misc{pith2026241118990,
author = {Pith},
title = {Pith review of: USTCCTSU at SemEval-2024 Task 1: Reducing Anisotropy for Cross-lingual Semantic Textual Relatedness Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/LETX3MUK}},
note = {Machine review of arXiv:2411.18990}
}
read the original abstract
Cross-lingual semantic textual relatedness task is an important research task that addresses challenges in cross-lingual communication and text understanding. It helps establish semantic connections between different languages, crucial for downstream tasks like machine translation, multilingual information retrieval, and cross-lingual text understanding.Based on extensive comparative experiments, we choose the XLM-R-base as our base model and use pre-trained sentence representations based on whitening to reduce anisotropy.Additionally, for the given training data, we design a delicate data filtering method to alleviate the curse of multilingualism. With our approach, we achieve a 2nd score in Spanish, a 3rd in Indonesian, and multiple entries in the top ten results in the competition's track C. We further do a comprehensive analysis to inspire future research aimed at improving performance on cross-lingual tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Zheng Lin Chia, Michal Ptaszynski, Fumito Masui, Gniewosz Leliwa, and Michal Wroczynski. 2021. Machine learning and feature engineering-based study into sarcasm and irony classification with application to cyberbullying detection. Information Processing & Management, 58(4):102600
work page 2021
-
[4]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, \'E douard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440--8451
2020
-
[5]
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. Advances in neural information processing systems, 32
2019
-
[6]
Cheng Fan, Yongjun Sun, Yang Zhao, Mengjie Song, and Jiayuan Wang. 2019. Deep learning-based feature engineering methods for improved building energy prediction. Applied energy, 240:35--45
work page 2019
-
[7]
Pablo Gamallo, Caroline Gasperin, Alexandre Agustini, and Gabriel P Lopes. 2001. Syntactic-based methods for measuring word similarity. In International Conference on Text, Speech and Dialogue, pages 116--125. Springer
work page 2001
-
[8]
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Representation degeneration problem in training natural language generation models. arXiv e-prints, pages arXiv--1907
work page 2019
Show all 30 references
-
[9]
Alex Graves and Alex Graves. 2012. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pages 37--45
2012
-
[10]
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2
2019
-
[11]
Anna Koroleva, Sanjay Kamath, and Patrick Paroubek. 2019. Measuring semantic similarity of clinical trial outcomes using deep pre-trained language representations. Journal of Biomedical Informatics, 100:100058
2019
-
[12]
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043
2017 arXiv
-
[13]
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541--551
1989
-
[14]
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119--9130
2020
-
[15]
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Met...
2020
-
[16]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
2019 arXiv
-
[17]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781
2013 arXiv
-
[18]
Leann Myers and Maria J Sirois. 2004. Spearman correlation coefficients, differences between. Encyclopedia of statistical sciences, 12
2004
-
[19]
Mohammad
Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Abinew Ali Ayele, Pavan Baswani, Meriem Beloucif, Chris Biemann, Sofia Bourhim, Christine De Kock, Genet Shanko Dekebo, Oumaim...
2024 arXiv
-
[20]
Mohammad
Nedjma Ousidhoum, Shamsuddeen Hassan Muhammad, Mohamed Abdalla, Idris Abdulmumin, Ibrahim Said Ahmad, Sanchit Ahuja, Alham Fikri Aji, Vladimir Araujo, Meriem Beloucif, Christine De Kock, Oumaima Hourrane, Manish Shrivastava, Thamar Solorio, Nirmal Surange, Krishnapriya Vishnub...
2024
-
[21]
Partha Pakray, Sivaji Bandyopadhyay, and Alexander Gelbukh. 2011. Textual entailment using lexical and syntactic similarity. International Journal of Artificial Intelligence and Applications, 2(1):43--58
2011
-
[22]
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543
2014
-
[23]
Jonathan Pilault, Amine Elhattami, and Christopher Pal. 2020. Conditionally adaptive multi-task learning: Improving transfer learning in nlp using fewer parameters & less data. arXiv preprint arXiv:2009.09139
2020 arXiv
-
[24]
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502
2019 arXiv
-
[25]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9
2019
-
[26]
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316
2021 arXiv
-
[27]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30
2017
-
[28]
Improving neural language generation with spectrum control
Lingxiao Wang, Jing Huang, Kevin Huang, Ziniu Hu, Guangtao Wang, and Quanquan Gu. Improving neural language generation with spectrum control
-
[29]
Sen Wu, Hongyang R Zhang, and Christopher R \'e . 2020. Understanding and improving information transfer in multi-task learning. arXiv preprint arXiv:2005.00944
2020 arXiv
-
[30]
Yin Zhang, Rong Jin, and Zhi-Hua Zhou. 2010. Understanding bag-of-words model: a statistical framework. International journal of machine learning and cybernetics, 1:43--52
2010
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.