{"id":"57573e2f-e50c-461d-a138-67c99e7fb5e0","arxiv_id":"2501.12336","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A shared-task entry combining fixed XLM-R sentence embeddings with a small regression network predicts annotator disagreement rankings, reaching average Spearman 0.124 and mid-pack standing.","lead":"This paper describes a system for ranking how much human annotators disagree about word meanings, using multilingual sentence embeddings and a small neural network. It reports the system's scores on the CoMeDi shared task, where it finished mid-pack among seven teams.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's headline claim of 'ranked 3rd among 7 teams' is contradicted by Table 3, where the system's average Spearman (0.124) carries rank (4); no tie-break or aggregation rule is given to resolve the discrepancy.","rationale":"The paper is a shared-task system report whose main contribution is an empirical result: a particular combination of frozen XLM-R sentence embeddings and a deep regression head ranks 3rd among 7 teams. The reader's verdict already flags this as conditional because of the table's average rank of 4 and ambiguous protocol. I agree that the rank discrepancy is material. My concern is narrower and more concrete: the claim of '3rd' is internally contradicted by Table 3's own average-rank cell, and no aggregation or tie-breaking rule is provided. This is a checkable factual inconsistency, not a theoretical weakness. The frozen-embedding concern raised by the reader is real but is a performance limitation, not a fatal flaw; even if Spanish and Chinese correlations are near zero, the system could still have placed 3rd on average. The decisive issue is whether the rank claim is true. The proposed check against the official leaderboard will settle it. Because the paper's value as a system description survives correction, I do not move the verdict; the same conditional acceptance with a required correction is appropriate.","tokens_in":5663,"tokens_out":4785,"duration_ms":43481,"concrete_test":"Obtain the official CoMeDi Subtask 2 evaluation leaderboard (from the shared task organizers or the workshop report, Schlechtweg et al. 2025) and reproduce the average Spearman and rank for each of the 7 teams using the same per-language scores and tie-breaking rule. If FuocChuVIP123 is officially ranked 4th, the Abstract, Section 6, and Section 7 must be corrected to say 4th; if officially 3rd, Table 3's average-rank cell and parenthetical per-language ranks must be corrected, and the aggregation formula should be stated. This check settles whether the central empirical claim is accurate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is the empirical result 'ranked 3rd among 7 teams' in CoMeDi Subtask 2 (Abstract; Section 6; Section 7). This claim is not supported by the paper's own Table 3: in the 'Evaluation' phase row for FuocChuVIP123, the average Spearman value is 0.124 with a parenthetical rank of (4), while per-language ranks are (4) for ZH, (7) for EN, (3) for DE, (4) for NO, (6) for RU, (7) for ES, and (3) for SV. No aggregation method or tie-breaking rule is described that would convert these values into an overall rank of 3. The phrase 'ranked 3rd out of 7 teams' is therefore an internal inconsistency, not merely a wording choice: if the official rank is 4, the paper overstates its result; if it is 3, Table 3's rank labels and the average column are wrong. Until the official leaderboard ranking is reconciled with the table, the 'competitive performance' claim cannot be evaluated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes FuocChuVIP123's submission to the CoMeDi Shared Task, Subtask 2 (DisWiC), which asks systems to rank word-in-context pairs by the mean pairwise disagreement of annotator judgments across seven languages. The system concatenates 768-dimensional sentence embeddings from the frozen paraphrase-xlm-r-multilingual-v1 model and trains a four-hidden-layer MLP with batch normalization, dropout, and an MSE loss to regress the mean disagreement score. Performance is measured by Spearman correlation against the official labels. The paper reports evaluation-phase scores in Table 3 and claims that the system achieved competitive performance, ranking 3rd among 7 teams.","tokens_in":5887,"tokens_out":3894,"duration_ms":38324,"significance":"If the reported evaluation-phase scores are accurate, the paper documents a simple and transparent baseline: frozen off-the-shelf multilingual sentence embeddings plus a small regression head. The main value is empirical: this approach yields moderate correlations for German and Swedish but near-zero correlations for Spanish and Chinese, which is a useful data point for shared-task participants. The architecture description is clear enough to reproduce. However, the interpretation is overstated: the average rank shown in the paper's own table is 4th, not 3rd, and the 'competitive performance' language is not supported by the near-random results on two languages. With a corrected and more carefully framed presentation, the paper could be an acceptable system-description contribution, but the current central claim needs to be reconciled with the data.","major_comments":[{"comment":"The headline claim that the system 'ranked 3rd out of 7 teams' in the evaluation phase is not supported by Table 3. The table lists FuocChuVIP123 with an average Spearman correlation of 0.124 and a parenthetical rank of (4), and per-language ranks of (4) for ZH, (7) for EN, (3) for DE, (4) for NO, (6) for RU, (7) for ES, and (3) for SV. No aggregation rule, tie-breaking rule, or official leaderboard excerpt is provided that would convert these values into an overall rank of 3. Please reconcile the stated rank with Table 3 or revise the Abstract and Conclusion accordingly.","section":"Abstract; §6; Table 3"},{"comment":"The training and evaluation description is ambiguous in a way that affects reproducibility and the validity of model selection. The text says 'We merged the necessary information from training and development sets to construct the input for our model' and later 'Our model was evaluated on the development set,' while §4.4 describes an 80/20 training/validation split. It is not clear whether the development set was used for training, for model selection, or only for final evaluation, or what exactly was merged. Please specify the exact data flow—which splits were used for fitting, early stopping, and reporting—and state whether any development-set labels influenced the final model.","section":"§5"},{"comment":"The explanation of the system's weak performance is inconsistent with Table 3. Section 8 says the system 'struggled with Latin-based languages like Spanish,' and Section 6 refers to challenges with 'Latin languagues,' but the two worst per-language correlations in Table 3 are Chinese (0.018) and Spanish (0.012), while English (0.078) also ranks last among the seven systems listed. The claim about language families is not supported by an analysis; either provide language-level diagnostics or withdraw the explanatory claim.","section":"§6; §8"}],"minor_comments":[{"comment":"The sample counts in Table 1 are printed as decimals (20.46, 10.83, 6.04, etc.), whereas Table 2 lists integers (871, 1,932, 1,269) and §3.1 says counts range from 1,222 to 24,891; please clarify the units or formatting (e.g., thousands) and make the two tables consistent.","section":"Tables 1 and 2"},{"comment":"There is a typo: 'languagues' should be 'languages.'","section":"§6"},{"comment":"The reference to (Mostafazadeh Davani et al., 2022) is duplicated in the same sentence; please cite the relevant work once and distinguish it from any other cited work by the same authors.","section":"§2"},{"comment":"The description of XLM-RoBERTa as ending with a sigmoid activation and cross-entropy loss describes a classification head that is not part of the regression system used in this paper; consider removing this passage or explicitly labeling it as background.","section":"§4.3"},{"comment":"The column header 'A VG' appears to be a typo for 'Avg,' and the average column should be defined (e.g., whether it is the mean of the per-language Spearman values).","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The rank discrepancy between the text ('3rd of 7 teams') and Table 3 (average rank 4) is the main obstacle to acceptance; the official CoMeDi leaderboard should be consulted to determine which number is correct. If the official rank is 4, the paper should be reframed as a system description without the 'competitive' claim. The ambiguous merging of training and development sets in §5 also needs a clear statement of the data flow."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear X,\n\nShort version: it's a competent shared-task system description with one internal inconsistency that needs correcting. The actual contribution is the reported scores on Subtask 2 – fresh empirical data on an externally evaluated benchmark. The method (frozen paraphrase-xlm-r-multilingual-v1 embeddings concatenated into an MLP regressor) is standard, but the task framing is right: regressing the mean pairwise annotation difference directly instead of a gold label.\n\nWhat the paper does well: it gives enough architectural and training detail to reproduce, it includes a limitations section that openly says Spanish and Chinese were near zero, and it doesn't oversell the approach itself. The related work is appropriate for a short system paper.\n\nThe soft spots are real but manageable. First, the abstract and conclusion say the team 'ranked 3rd among 7 teams,' but Table 3 shows the average Spearman of 0.124 with a rank of (4) in that table. No aggregation rule or tie-break is stated to square these. That's a load-bearing inconsistency in the paper's own numbers, and the author needs to resolve it before the claim is credible. Second, 'competitive performance' is an overstatement given correlations of 0.012 (Spanish) and 0.018 (Chinese), though the limitations section partially covers this. Third, Section 5 says 'We merged the necessary information from training and development sets' – that is ambiguous about whether development labels entered training, which matters for evaluating the official scores. Finally, there's no analysis of why the frozen embeddings fail so badly for those two languages, which is the most interesting question the paper raises. These are all fixable in revision.\n\nOverall, the paper is honest, clearly written, and reports externally produced scores. It deserves a serious referee – mostly to force the author to sort out the rank claim and clarify the training protocol. I wouldn't cite it in my own work, but if you're tracking shared-task results on disagreement prediction, it's worth a look.\n\nYes, send to peer review if the venue wants shared-task system descriptions; just expect the revision to tighten the claims.","headline":"A modest shared-task system report with real empirical scores but an internally inconsistent 'ranked 3rd' claim that needs fixing.","tokens_in":6400,"tokens_out":3059,"would_cite":false,"duration_ms":28165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen multilingual sentence encoder feeding a deep regression head can rank word-use pairs by annotator disagreement, and the resulting system finished third among seven teams in CoMeDi Subtask 2.","keywords":["disagreement ranking","word-in-context","multilingual sentence embeddings","XLM-RoBERTa","deep regression","Spearman correlation","shared task","annotation disagreement"],"falsifier":"Replacing the deep MLP with a plain linear regression on the same 1,536-dimensional concatenated embeddings would settle whether the architecture contributes anything: if linear regression matches or beats the reported Spearman values on each language, the ranking signal lives in the embeddings, not the deep head. Conversely, fine-tuning only the sentence encoder on the mean-disagreement objective and observing whether the Spanish (0.012) and Chinese (0.018) correlations recover would test whether the frozen-embedding choice is the main bottleneck.","tokens_in":5451,"feed_emoji":"📊","tokens_out":9951,"duration_ms":90144,"temperature":0.7,"pith_summary":"This system paper claims that a simple pipeline—concatenated sentence embeddings from a frozen multilingual transformer, passed through a deep regression head—can rank word-use pairs by how much annotators disagree, and that this pipeline placed third among seven teams on CoMeDi Subtask 2. The target is deliberately not a consensus label: the model is trained to predict the mean of pairwise absolute differences between annotator judgments, and evaluated with Spearman's rank correlation. On the hidden evaluation phase the system achieved an average Spearman correlation of 0.124, with the strongest per-language score for Swedish (0.172) and near-zero scores for Spanish (0.012) and Chinese (0.018). If the claim holds, it means useful cross-lingual disagreement ranking can be obtained without fine-tuning the encoder, using off-the-shelf multilingual sentence representations plus a comparatively small regression network.","feed_headline":"Off-the-shelf embeddings rank third in multilingual disagreement task","feed_subtitle":"Frozen XLM-R sentence embeddings plus a deep regressor predict annotator disagreement without fine-tuning the encoder.","key_machinery":"The load-bearing object is the concatenated embedding pair $X=[E(C_1),E(C_2)]$, where each $E(C)$ is the 768-dimensional vector from the frozen paraphrase-xlm-r-multilingual-v1 sentence encoder built on XLM-RoBERTa. The regression head maps this vector through four fully connected layers with widths 512, 256, 128, and 64, each followed by batch normalization, ReLU, and dropout at rate 0.3, then to a single output trained with mean squared error. This mechanism converts a pair of contexts into a scalar disagreement score without any task-specific tuning of the underlying encoder, so the ranking signal must come from the pretrained embeddings plus the learned head.","core_discovery":"The paper's central claim, stated on its own terms, is that disagreement ranking in a multilingual word-in-context setting reduces to a regression problem on concatenated contextual embeddings. For each word-use pair, the system takes the 768-dimensional sentence embedding of each context from paraphrase-xlm-r-multilingual-v1, concatenates them into a 1,536-dimensional vector, and trains a four-hidden-layer feedforward network with batch normalization and dropout to predict the mean pairwise absolute difference of the annotators' ordinal judgments. The paper reports that this explicit modeling of disagreement, rather than aggregation toward a gold label, produced competitive Spearman correlations and ranked third of seven teams in the evaluation phase. The same setup is offered as evidence that robust multilingual embeddings and a regularized deep regression head are sufficient ingredients for this task.","pith_inferences":["A natural test the paper does not run is to compare against a linear regression on the same concatenated embeddings; if the linear model matches the deep head's Spearman values, the reported success would be attributable to the embeddings rather than the architecture.","The near-zero Spanish and Chinese scores suggest that the sentence-level pooling may discard the target-word information that disagreement depends on; a token-level pooling around the target word indices could be investigated as a fix.","One could extend the evaluation to per-language Spearman values across multiple training seeds, since the paper reports no variance and it is unclear whether the 0.012 and 0.018 scores are stable or noise.","Because the encoder is frozen, the system leaves on the table any gains from task-adaptive training; an immediate extension is to fine-tune a lightweight adapter or the upper transformer layers on the mean-disagreement objective."],"forward_implications":["Other teams can reproduce a competitive multilingual disagreement ranker by combining a publicly available sentence encoder with a small regularized regression head, without fine-tuning the encoder.","Because the regression target is the mean pairwise absolute judgment difference, the same architecture could be retrained on other annotation tasks where disagreement, not consensus, is the quantity of interest.","The per-language evaluation results imply that frozen multilingual embeddings transfer unevenly: languages with longer or typologically different contexts, such as Spanish, may need language-specific adaptation to avoid near-zero correlation.","The system's third-place ranking in a hidden evaluation phase suggests that simpler embedding-plus-regression baselines can serve as a strong reference point for future disagreement-ranking systems."],"supporting_citations":[{"why":"Defines CoMeDi Subtask 2 and its mean-disagreement ranking target, evaluated with Spearman correlation.","marker":"(Schlechtweg et al., 2025)"},{"why":"Supplies the XLM-RoBERTa base model from which the paraphrase sentence encoder is built.","marker":"(Conneau et al., 2020)"},{"why":"Provides the DWUG word-usage datasets in multiple languages used for training and evaluation.","marker":"(Schlechtweg et al., 2024)"},{"why":"Defines the Spearman rank correlation coefficient used as the task metric.","marker":"(Zar, 2005)"},{"why":"Motivates preserving disagreement information instead of aggregating to a single label, the premise behind the prediction target.","marker":"(Uma et al., 2021)"}],"fun_headline_variants":["Disagreement ranking as regression on frozen sentence embeddings","Concatenate XLM-R embeddings and regress to rank disagreement","Third place for regression on embeddings over gold-label aggregation","Deep regressor on frozen embeddings predicts annotator disagreement","Frozen embeddings plus deep regressor rank third in CoMeDi task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen sentence encoder's embeddings carry enough signal about a word-use pair to predict how much annotators will disagree; if that premise fails, the whole pipeline fails regardless of the regression head.","fun_headline_variants_meta":{"raw":{"variants":["Disagreement ranking as regression on frozen sentence embeddings","Concatenate XLM-R embeddings and regress to rank disagreement","Third place for regression on embeddings over gold-label aggregation","Deep regressor on frozen embeddings predicts annotator disagreement","Frozen embeddings plus deep regressor rank third in CoMeDi task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000408,"raw_usage":{"total_tokens":2074,"prompt_tokens":858,"completion_tokens":1216,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":1133}},"tokens_in":474,"tokens_out":1216,"duration_ms":11965,"temperature":1.0,"reasoning_tokens":1133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:14:57.300649+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replacing the deep MLP with a plain linear regression on the same 1,536-dimensional concatenated embeddings would settle whether the architecture contributes anything: if linear regression matches or beats the reported Spearman values on each language, the ranking signal lives in the embeddings, not the deep head. Conversely, fine-tuning only the sentence encoder on the mean-disagreement objective and observing whether the Spanish (0.012) and Chinese (0.018) correlations recover would test whether the frozen-embedding choice is the main bottleneck.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates preserving disagreement information instead of aggregating to a single label, the premise behind the prediction target."}],"review_version":1}