{"id":"4fad5a5b-c7e2-4fcf-b7d5-dc50a2478e21","arxiv_id":"2411.18990","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Applying whitening to XLM-R-base sentence embeddings, together with a language-filtering heuristic, improved cross-lingual relatedness scores and placed second in Spanish and third in Indonesian in SemEval-2024 Task 1 Track C.","lead":"A competition entry for a cross-lingual semantic relatedness task uses a multilingual transformer plus a whitening transform to spread out sentence vectors, along with a heuristic to drop training languages that hurt the target language. The system placed second in Spanish and third in Indonesian in the SemEval-2024 Track C leaderboard.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Whitening gain is not isolated: whitening parameters are fit to target test text and Table 1 mixes whitening with data filtering, so the reported improvements may be transductive artifacts rather than evidence for the anisotropy story.","rationale":"The reader's weakest assumption identifies the transductive use of target test text for whitening, and that is indeed a central problem. I partially agree but sharpen the concern: the paper's own Section 3.4 and Table 1 show that the final Spanish score uses data filtering, so the row labeled 'XLM-Rbase-whitening' is not a pure whitening ablation. Thus even within the transductive setting, the contribution of whitening is not identified. The rankings are externally verified, so as a system description the empirical results are credible, but the scientific claim that whitening reduces anisotropy and improves cross-lingual relatedness is under-supported. The proposed controlled ablation would settle the attribution. Since the reader already recommended a conditional verdict with disclosure and ablations, my read does not change the verdict; I would keep it conditional pending the ablation evidence.","tokens_in":7439,"tokens_out":6020,"duration_ms":58125,"concrete_test":"Run the following controlled ablation on the official ind and esp test sets: (1) XLM-Rbase, no whitening, no filtering; (2) XLM-Rbase + whitening with mu/W estimated from source-language training text only (no target test text); (3) XLM-Rbase + whitening with mu/W estimated from target test text as in the paper; (4) each condition with and without the data-filtering training-set selection. If condition (2) gives no significant gain over (1), while (3) matches Table 1, the reported whitening benefit is a test-set-fitting artifact. Also report condition (1) + filtering to isolate the filtering contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing issue is attribution. Section 4 states that 'we compute the mu and W of whitening using the text data of the target languages in track C' and uses this target-test-derived whitening both in the final system and in the per-language filtering decision. The baseline 'without whitening' uses no test-distribution information, so the comparison in Table 1 does not isolate the whitening transformation; it also includes (a) access to unlabeled target test sentences and (b) the data-filtering step. The Spanish number in Table 1 (0.6886) is exactly the value reported in Section 3.4 for 'training set with data filtering', confirming that the row labeled XLM-Rbase-whitening includes filtering. No condition is reported for XLM-Rbase + filtering without whitening, or for whitening estimated from source-language or development text. Because the whitening transform is fit to the same test distribution on which it is evaluated, the improvement could be a transductive domain-adaptation effect, not evidence that whitening reduces anisotropy in a way that transfers to new target languages. The paper's central claim therefore rests on an unestablished identification of which component (whitening, filtering, or test-input access) produces the observed Spearman gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the USTCCTSU submission to SemEval-2024 Task 1 Track C, a cross-lingual semantic textual relatedness task. The system freezes XLM-R-base, applies a whitening transformation to the resulting sentence embeddings to reduce anisotropy, and uses a language-filtering method that selects training languages based on the target language's unlabeled text. The authors report Spearman correlations on the Indonesian and Spanish test sets, claiming a 2nd-place result in Spanish and 3rd in Indonesian, and attribute the gains to whitening and data filtering. The paper also includes a cosine-similarity distribution analysis intended to show that whitening makes the embedding space more isotropic.","tokens_in":7705,"tokens_out":4047,"duration_ms":39691,"significance":"The competition results are externally supported: the reported ranks in Track C are verifiable leaderboard outcomes, and the paper provides a clear presentation of the whitening formulas and the system configuration. If the causal claims were established, the work would be a useful case study in applying whitening and training-set selection to cross-lingual semantic relatedness. However, the current evidence does not isolate the effect of whitening from the data-filtering step or from access to the target-language test distribution, and no statistical reliability checks are reported. The significance is therefore conditional on a revision that provides ablations and clarifies the transductive nature of the procedure.","major_comments":[{"comment":"The attribution of the gain to whitening is confounded. The Spanish value reported for XLM-Rbase-whitening in Table 1 (0.6886) is exactly the value reported in Section 3.4 for the training set with data filtering, so the row labeled 'XLM-Rbase-whitening' includes the data-filtering step. The comparison against the baseline therefore conflates at least three factors: the whitening transform, the use of target-language test data to estimate the transform, and the language filtering. No condition is reported for the model with filtering but without whitening, nor for whitening whose parameters are estimated from source-language or development data. The central claim that whitening itself improves cross-lingual semantic relatedness cannot be evaluated from the presented table.","section":"Section 4, Table 1"},{"comment":"The whitening parameters mu and W are computed using the target-language text from Track C, and the same target-language test data are then used to compute the reported Spearman correlations. This is a transductive setup: the representations are fitted to the test distribution on which they are evaluated. Consequently, the reported scores do not demonstrate that the whitening step transfers to new target languages or that the observed gains would persist under a distribution shift. If the competition rules permit the use of unlabeled test sentences, this should be stated explicitly and its implications for the generalization claim should be discussed; otherwise, the experimental protocol does not support the paper's conclusions.","section":"Section 4"},{"comment":"The data-filtering method rests on an unvalidated symmetry assumption: the paper asserts that if including target-language text degrades prediction on a training language, then that training language degrades prediction on the target language. No evidence is provided for this reversibility. Moreover, the filtering decision is made using whitening parameters computed from the target test text, so the selected training languages are chosen specifically for the whitening transform that will be used on that test set. A held-out validation step is needed to show that the filtering procedure improves target-language performance rather than merely selecting languages that work well with a test-set-specific transform.","section":"Section 3.4"},{"comment":"The reported improvements lack statistical reliability measures. The Indonesian gain from baseline to XLM-Rbase-whitening is 0.4746 versus 0.4700, a difference of 0.0046, and no error bars, repeated runs, or significance tests are reported. Given that the whitening row also includes filtering and test-distribution access, this difference is far too small to support the claim that whitening 'significantly improves' the task. The authors should report variance across runs or at minimum justify that the differences exceed the noise level of the evaluation.","section":"Table 1, Section 5"}],"minor_comments":[{"comment":"The derivation of the whitening transform largely follows Su et al. (2021), but the notation is occasionally inconsistent: for example, equation (1) uses x_i while s_i was introduced as the sentence vector. Please align the notation throughout.","section":"Section 3.3"},{"comment":"The paper does not explain what the 'topk parameter of whitening' means or how the value 256 was chosen. Since whitening normally uses the full covariance matrix, the truncation to 256 components and its effect on results should be clarified.","section":"Section 4"},{"comment":"The description of the filtering example for Spanish is confusing: the phrase '1,000 each of all data except Spanish' is followed by '1,000 each of kin and ind', but it is not clear whether the final Spanish training set contains only two languages or whether other languages were considered and excluded. Please specify the exact language sets before and after filtering.","section":"Section 3.4"},{"comment":"Figure 2 does not specify which language or data split the cosine-similarity histograms are computed on, nor whether the same sentence pairs are used in both panels. Adding this information would make the anisotropy claim easier to verify.","section":"Figure 2"},{"comment":"There are numerous typographical and formatting issues, including 'XLM -Rbase' with inconsistent spacing, 'RoBERT' instead of 'RoBERTa', 'are were significantly improved', 'We verifies', and an incomplete reference entry for 'Wang et al.'. A careful proofreading pass is needed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The leaderboard results are credible, but the manuscript's scientific contribution is currently a system description with causal claims that outrun the evidence. The required ablations and a clear statement about the transductive use of test data are feasible within the scope of a revision, so I recommend major revision rather than rejection. If the venue expects a substantive research contribution beyond a competition report, the authors should also consider whether the paper's length and novelty meet that bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing in this paper is a genuine competition result: XLM-R-base with whitening and a language-filtering heuristic took second in Spanish and third in Indonesian on SemEval-2024 Task 1 Track C. That ranking is externally verified and worth recording.\n\nWhat's actually new: applying Su et al.'s whitening to cross-lingual semantic relatedness, plus a heuristic for dropping training languages based on cross-effects measured through the target-language text. The paper is honest enough to state in Section 4 that the whitening mean and covariance are computed from the target-language test data, and that the filtering decisions are made with that same text.\n\nThe soft spots are real, and they are about attribution. Table 1 labels rows 'with whitening,' but the Spanish value 0.6886 is exactly the number reported in Section 3.4 for 'training set with data filtering.' So the 'XLM-Rbase-whitening' row includes the filtering step; there is no condition for whitening alone, nor for filtering alone. The gain attributed to whitening is therefore not isolated. Add the transductive setup: the whitening parameters are fit to the same test sentences being scored, while the baseline has no access to test text. The improvement could be a test-time adaptation effect, not a demonstration that whitening makes embeddings isotropic in any transferable sense. The Indonesian gain is 0.0046, well within noise, and there are no error bars or significance tests anywhere. The symmetry assumption behind the filtering rule ('if A hurts B, then B hurts A') is asserted, not tested.\n\nNone of this kills the paper as a competition report. The method is coherent, the data filtering gave a clear gain for Spanish (0.6375 to 0.6886), and the cosine-distribution plots in Figure 2 do show the anisotropy problem the authors describe. But the paper overclaims when it says whitening 'significantly improves' performance; the current experiments cannot support that.\n\nThis is a paper for the SemEval-style audience and for anyone working on multilingual sentence embeddings who wants to see a concrete transductive pitfall. With ablations separating whitening from filtering, and a whitening condition fit on non-test text, it would be worth a serious referee. As written, it needs that revision.\n\nRecommendation: engage with it, but send it back for major revision with the ablations required. It should not be desk-rejected; the competition result is real and the confounded analysis is fixable.","headline":"Competition result is real, but the paper's attribution of the gain to whitening is confounded by the data-filtering step and by fitting whitening parameters to the test set itself.","tokens_in":8244,"tokens_out":3355,"would_cite":false,"duration_ms":28272,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding a whitening transform to XLM-RoBERTa-base sentence embeddings, together with a language-level training-data filter built from unlabeled target-language text, is what produces top rankings in cross-lingual…","keywords":["cross-lingual semantic textual relatedness","anisotropy reduction","sentence embedding whitening","XLM-RoBERTa","multilingual data filtering","curse of multilingualism","Spearman correlation","SemEval-2024 Task 1"],"falsifier":"Estimate the whitening mean and covariance from target-language test-distribution text versus only from source-language text, run the same frozen encoder and filtering rule under both conditions, and compare the final Spearman correlations; if the gap nearly vanishes, the anisotropy fix is not the main driver of the reported leaderboard results.","tokens_in":7217,"feed_emoji":"🌐","tokens_out":6611,"duration_ms":53993,"temperature":0.7,"pith_summary":"This paper targets cross-lingual semantic textual relatedness, where a system must score how related two sentences are even when its labeled training data come from other languages. It claims that two cheap post-processing choices account for its high rankings in SemEval-2024 Task 1 Track C: applying a whitening transform to XLM-RoBERTa-base sentence vectors to remove their anisotropic concentration, and filtering the multilingual training data language-by-language using unlabeled target-language text. With these choices, the system reports Spearman correlations of 0.6886 for Spanish and 0.4746 for Indonesian, placing second and third among submitted systems. The paper further verifies that whitening spreads the cosine-similarity distribution from a narrow band near 1.0 toward a more normal-shaped spread, which is the mechanism it credits for the gains.","feed_headline":"Whitening lifts cross-language text relatedness to 2nd, 3rd","feed_subtitle":"Adding a whitening step to XLM-R vectors plus language-data filtering places 2nd in Spanish, 3rd in Indonesian.","key_machinery":"The carrying object is the whitening transform $\\tilde{s}_i = (s_i - \\mu) W$, with $\\mu$ the mean vector and $W = U\\Lambda^{-1/2}$ obtained from the singular value decomposition of the covariance matrix $\\Sigma = U\\Lambda U^\\top$; this maps the sentence-vector cloud to zero mean and identity covariance. It is applied to frozen XLM-RoBERTa-base representations, with a top-$k$ cutoff of 256. The second module is a data-selection rule: using unlabeled target-language text to compute the whitening parameters, the paper computes the Spearman correlation for each candidate training language, keeps languages whose inclusion improves prediction, and drops those that hurt, on the assumption that language influence is mutual.","core_discovery":"The central claim is that the poor cosine-similarity behavior of multilingual sentence encoders is fixable at inference time, without fine-tuning, by whitening the embedding space, and that training-set composition can be improved by deleting languages that hurt rather than help. Concretely, the paper shows that XLM-RoBERTa-base with whitening outperforms the same encoder without it (Indonesian 0.4746 vs 0.4390; Spanish 0.6886 vs 0.5907), and that dropping most training languages in favor of two better-matched languages raises Spanish performance from 0.6375 to 0.6886. The system achieves 2nd place in Spanish and 3rd in Indonesian in Track C of SemEval-2024 Task 1.","pith_inferences":["The reported rankings assume the system can see unlabeled text from the exact target language before scoring; in a deployment where no such text exists, the whitening step would need a proxy corpus, and the gains could shrink.","The data-filtering heuristic is separable from whitening and could be tested as a standalone training-set selection method for other multilingual transfer tasks.","The fixed top-$k$ of 256 suggests that the choice of how many whitened dimensions to keep may matter as much as the whitening itself, inviting experiments with a learned rank rather than a fixed cutoff."],"forward_implications":["Whitening is a no-fine-tuning post-processing step that can be dropped into frozen multilingual encoders to improve relatedness scoring.","Filtering the training data by measured language influence lets a system use fewer languages and obtain better results, easing the curse of multilingualism.","The measured cosine-similarity shift from a narrow concentrated band to a wider distribution indicates that anisotropy, not model capacity alone, was limiting the baseline.","The approach transfers to target languages without requiring labeled target-language data, needing only unlabeled text from the target language.","The same whitening-plus-selection recipe can be evaluated on other cross-lingual tasks where embeddings are compared by cosine similarity."],"supporting_citations":[{"why":"Supplies the whitening transform and its derivation that the paper applies to XLM-R sentence vectors.","marker":"(Su et al., 2021)"},{"why":"Provides the XLM-RoBERTa-base encoder and documents the curse of multilingualism that motivates the data-filtering module.","marker":"(Conneau et al., 2020)"},{"why":"Provides the labeled relatedness datasets in 14 languages that form the training data and the baseline the paper beats.","marker":"(Ousidhoum et al., 2024a)"},{"why":"Defines Task 1 Track C of SemEval-2024, the evaluation protocol, and the leaderboard against which rankings are reported.","marker":"(Ousidhoum et al., 2024b)"},{"why":"Establishes the anisotropy of BERT-family sentence embeddings that the paper identifies as the obstacle and targets with whitening.","marker":"(Li et al., 2020)"},{"why":"Documents the representation degeneration problem that underlies the anisotropic vector space argument.","marker":"(Gao et al., 2019)"}],"fun_headline_variants":["Whitening and data filtering land 2nd in Spanish, 3rd in Indonesian","Whitening XLM-R vectors without fine-tuning earns top SemEval ranks","Data filtering and whitening push XLM-R to top-3 in SemEval","Inference-time whitening boosts cross-lingual relatedness by up to 10 pts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the system may compute the whitening mean and covariance from unlabeled text in the exact target language at test time; remove that access and the method as described cannot be applied.","fun_headline_variants_meta":{"raw":{"variants":["Whitening and data filtering land 2nd in Spanish, 3rd in Indonesian","Whitening XLM-R vectors without fine-tuning earns top SemEval ranks","Data filtering and whitening push XLM-R to top-3 in SemEval","Inference-time whitening boosts cross-lingual relatedness by up to 10 pts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001102,"raw_usage":{"total_tokens":4558,"prompt_tokens":866,"completion_tokens":3692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":3602}},"tokens_in":482,"tokens_out":3692,"duration_ms":21453,"temperature":1.0,"reasoning_tokens":3602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:39:04.685142+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate the whitening mean and covariance from target-language test-distribution text versus only from source-language text, run the same frozen encoder and filtering rule under both conditions, and compare the final Spearman correlations; if the gap nearly vanishes, the anisotropy fix is not the main driver of the reported leaderboard results.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the representation degeneration problem that underlies the anisotropic vector space argument."}],"review_version":1}