{"id":"b3d7b7ba-e28e-4b44-8493-9b714eee3586","arxiv_id":"2412.11750","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A confidence-based variant of Datamaps ranks common Spanish variety examples above random baseline, and a new 1,762-tweet Cuban Spanish dataset is presented.","lead":"This paper tests whether a model's changing confidence during training can flag Spanish sentences that are valid in more than one dialect, called common examples. It also introduces a new manually annotated dataset of Cuban Spanish tweets, the first resource focused on a Caribbean Spanish variety.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The method may be detecting generic difficulty rather than commonness; the random label assignment in Section 4.2 makes the Cuban evaluation unable to separate label noise from commonness.","rationale":"The paper's central claim is that tracking a binary classifier's confidence in its predicted label can rank human-annotated common examples above a random ranking. The DSL-TL experiment provides independent support for this: common examples there carry original source labels rather than randomly assigned ones, and DMmean-pred still outperforms random. That is genuine evidence and should be credited. The load-bearing concern is that the method's signal is assumed to be commonness-specific, but it may instead be a generic hardness signal. The Cuban dataset's random-label simulation directly confounds 'common' with 'noisy', and the paper's own error analysis shows topic bias, so the interpretation of the ranking results as detecting commonness is not secure. This concern does not overturn the paper—the DSL-TL result stands—but it does mean the cross-domain claim and the value of the new dataset need a control experiment before they are fully supported. The reader's conditional verdict is appropriate; my analysis reinforces it without moving the verdict.","tokens_in":12882,"tokens_out":7199,"duration_ms":68244,"concrete_test":"Run a matched control on CUBAN SPVARIETY: take the non-common examples (full annotator agreement) and randomly flip their binary labels for the same proportion as the ES set (46%), train a fresh BETO model, and compute DMmean-pred rankings. If the flipped non-common examples achieve APS and Top-N precision comparable to the true ES examples, the score is detecting manufactured label noise rather than commonness. Repeat across the five seeds used in the paper and report the comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 randomly assigns each common example (ES) in CUBAN SPVARIETY to either ES-CU or not-ES-CU before training. This manufactures label noise that is uncorrelated with linguistic commonness, and DMmean-pred—which scores low confidence in the predicted label—will inevitably flag these randomly labeled examples as hard. The evaluation therefore cannot tell whether the method detects commonness or simply detects the injected noise. The DSL-TL experiment is cleaner because its binary labels come from the original source corpus, and it provides real support for the ranking claim. However, the cross-domain claim (RQ3) depends on the Cuban dataset, and the paper's own Section 6 error analysis shows that topic words such as 'Cuba' and 'SOSCuba' dominate model predictions. The SHAP analysis of the lowest-ranked common examples (Figure 9b) shows they are confidently classified on topic or non-linguistic grounds, meaning the method misses common examples that the model confidently (but biasedly) labels. Together these observations indicate that the core assumption in Section 3.1—that low confidence in the predicted label is driven by a sentence being valid in multiple varieties—is not isolated from other sources of difficulty (label noise, topic bias, annotation disagreement). The reported gains over a random baseline are consistent with a generic hardness detector, not specifically a common-example detector.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a training-dynamics-based method for automatically detecting common examples (texts valid across multiple Spanish varieties) in variety-identification datasets. It adapts the Datamaps confidence and variability metrics to use the maximum predicted probability rather than the gold-label probability (Equations 1 and 2), and evaluates the resulting scorers (DMmean-pred and DMstd-pred) against a random baseline on two datasets: the Spanish subset of DSL-TL and a newly introduced CUBAN SPVARIETY dataset of 1,762 manually annotated tweets. The authors report that DMmean-pred outperforms the random baseline and DMstd-pred on both datasets, claim that this demonstrates the usefulness of predicted-label confidence for detecting common examples, and provide a SHAP-based error analysis. They also present CUBAN SPVARIETY as the first dataset for Cuban (Caribbean) Spanish variety identification.","tokens_in":13194,"tokens_out":3594,"duration_ms":32498,"significance":"If the central claim holds, the paper would provide a practical, low-cost tool for flagging ambiguous examples in variety-identification datasets, which is a real need given the high prevalence of common examples in Spanish and other languages. The new CUBAN SPVARIETY dataset is a potentially valuable resource, particularly because Caribbean Spanish varieties are underrepresented in NLP resources. The paper also makes a useful methodological contribution by distinguishing confidence-based from variability-based training-dynamics scorers and by performing a detailed error analysis that exposes topic bias. However, the evaluation design has two serious confounds that currently prevent the results from supporting the central claim, as detailed in the major comments: the random relabeling of common examples in the Cuban dataset and the conflation of 'both' and 'neither' labels in DSL-TL.","major_comments":[{"comment":"The random assignment of common examples to either ES-CU or not-ES-CU before training (Section 4.2) injects label noise that is uncorrelated with linguistic commonness. Since DMmean-pred scores examples by low confidence in the predicted label, it will systematically flag the randomly relabeled common examples as hard, regardless of whether their language is genuinely ambiguous. The evaluation therefore cannot separate detection of commonness from detection of injected noise, and the reported gains on CUBAN SPVARIETY (Table 1) may be an artifact of this procedure. The paper acknowledges the 'increased ambiguity' (Section 5.2) but does not treat this as a threat to validity. This is load-bearing for RQ3, which claims cross-domain effectiveness. I recommend either removing the random-assignment step and evaluating on the original annotations (e.g., as a multi-label or three-class problem), or comparing against a control condition in which non-common examples are also randomly relabeled to show that the method detects commonness, not label noise.","section":"Section 4.2, Section 5.2"},{"comment":"The DSL-TL 'common examples' class appears to merge the 'both' and 'neither' annotations from the original DSL-TL dataset. The paper states that 'a third label—both or neither—was introduced' and later treats the resulting 'ES' class as common examples. 'Neither' examples are not valid across multiple varieties; they are examples that annotators could not attribute to either variety, which is a different phenomenon. Including them in the common-example class conflates ambiguity from overlap with ambiguity from lack of identifiable features, and it is unclear how much of the reported DSL-TL performance is driven by each. The authors should separate these two cases or justify why they belong in the same class; without this, the DSL-TL results do not cleanly measure common-example detection.","section":"Section 4.1, Figure 2(a)"},{"comment":"The paper's own error analysis shows that topic words, especially 'Cuba' and 'SOSCuba', dominate the model's mistakes (about 67% of the top-500 errors contain the word 'Cuba', versus 33% in the whole dataset). This indicates that the low-confidence signal captured by DMmean-pred can reflect topic bias rather than genuine multi-variety validity, and the SHAP analysis of the lowest-ranked common examples (Figure 9b) shows examples confidently classified on topic or non-linguistic grounds. These observations directly challenge the core assumption in Section 3.1 that low confidence in the predicted label is driven by a sentence being valid in multiple varieties. To support the central claim, the authors should control for topic effects (e.g., by masking or replacing named entities and hashtags) and show that the ranking advantage persists on topic-balanced subsets.","section":"Section 6, Figure 7"},{"comment":"The paper claims that 'the two Datamaps models significantly outperform the baseline' and that DMmean-pred 'consistently outperforms' DMstd-pred, but no significance tests are reported. Given the reported standard deviations (e.g., DMmean-pred 54.75 ± 1.8 vs. DMstd-pred 52.88 ± 3.00 on DSL-TL; 63.51 ± 2.56 vs. 61.97 ± 2.60 on CUBAN SPVARIETY), some differences may not be significant across the five seeds. The authors should provide paired significance tests (e.g., bootstrap or permutation tests over the ranked lists) or at least report per-seed results and effect sizes, so that the 'significantly outperform' claim is backed by evidence.","section":"Table 1, Section 5.2"}],"minor_comments":[{"comment":"Table 2 is captioned 'DSL-TL Overview' but reports statistics (#sentences 1762, #tokens 41374) that match the CUBAN SPVARIETY dataset rather than DSL-TL. This is likely a labeling error and should be corrected.","section":"Appendix A, Table 2"},{"comment":"The annotation description says 'not_able_to_identify' but the guidelines in Appendix B use 'unable_to_identify_variety'; the terminology should be harmonized.","section":"Section 4.2"},{"comment":"The Limitations section acknowledges the binary-classification focus and the single-region annotator pool, but it does not mention the random relabeling of common examples or the both/neither conflation in DSL-TL; these should be listed as limitations given their impact on the reported results.","section":"Section 8"},{"comment":"The sentence 'The performance difference between DMmean-pred and DMstd-pred is more pronounced for smaller values of N, particularly in precision' is based on Figures 5 and 6, but those figures are not referenced at the point of the claim; adding explicit references would improve readability.","section":"Section 5.2"},{"comment":"The reference to 'Vaidya et al., 2024' in Section 1 is cited for language-specific models being more sensitive to regional variations, but the cited paper is about emotion detection in Hinglish and does not appear to support this specific claim; consider replacing with a more directly relevant citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real and timely problem, and the new dataset is a meaningful contribution. However, the evaluation currently has two confounds—the random relabeling in the Cuban experiment and the both/neither conflation in DSL-TL—that prevent the results from validating the central claim. These are fixable in principle (e.g., re-analyzing the original annotations, separating the DSL-TL labels, and adding topic controls), but they require substantial additional work. I would support a major-revision decision and encourage the editor to seek a revised version that either strengthens the evaluation or narrows the claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee, but the strongest result is narrower than the abstract suggests. What works: using the mean confidence in the predicted label over training epochs (a small but sensible Datamaps modification) ranks human-annotated common examples above a random baseline on DSL-TL by a wide margin, and the common labels there come from an external third class, independent of the binary training target. That is a clean, reproducible result. The new Cuban Spanish Twitter dataset is also a genuine contribution: first for Cuban and Caribbean variety identification, with three native annotators and agreement statistics reported.\n\nThe soft spots are real and roughly match the stress-test concern. In Section 4.2, common examples in the Cuban dataset are randomly assigned to ES-CU or not-ES-CU before training. DMmean-pred scores low confidence in the predicted label, so it will inevitably flag these randomly labeled examples as hard. The Cuban evaluation therefore cannot tell whether the method detects commonness or simply detects the injected label noise. This is disclosed in the text, but it still guts the cross-domain claim (RQ3). The DSL-TL experiment stands on its own, but the Cuban numbers should not be read as independent confirmation.\n\nTwo smaller issues. First, the DSL-TL third label mixes \"both\" and \"neither,\" so the positive class is not purely common examples; this likely dilutes precision rather than inflating it, but it should be acknowledged in the analysis. Second, the paper's own error analysis shows strong topic bias: 67% of top-500 errors in the Cuban set contain \"Cuba\" versus 33% of all examples, and the SHAP examples show confident topic-based classification. So \"low predicted confidence\" may be a generic hardness signal, not specifically multi-variety validity. That is fine as a practical heuristic, but the paper should not claim it isolates commonness.\n\nThe abstract also says the method \"enhances model performance in variety identification tasks,\" but no held-out or downstream experiment supports that. The appendix benchmarks are useful but not tied to the detection pipeline. The dataset and code are promised but not yet released, so the reproducibility evidence is currently limited to the reported numbers.\n\nOverall: a solid DSL-TL result, a valuable dataset, and an honest limitations section. The paper needs revision before publication: fix or reframe the Cuban evaluation, add a non-random baseline (e.g., gold-label Datamaps or a lexical heuristic), and soften the performance claim. Send it to peer review rather than desk-rejecting.","headline":"The DSL-TL half gives real evidence that predicted-label confidence ranks common examples above random, and the Cuban dataset is a genuine first, but the Cuban evaluation conflates commonness with injected label noise and the paper overclaims downstream benefit.","tokens_in":13642,"tokens_out":2389,"would_cite":false,"duration_ms":26210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A classifier's confidence in its predicted label can surface Spanish sentences valid in more than one variety.","keywords":["Spanish varieties","language variety identification","common examples","training dynamics","Cuban Spanish","dataset annotation","confidence scoring","NLP fairness"],"falsifier":"Recompute the DMmean-pred ranking on a version of CUBAN SPVARIETY where the random binary labels assigned to common examples are flipped; if the precision of the top-500 ranked common examples stays essentially unchanged, the score is tracking the manufactured label noise, not the commonness, and the central claim fails.","tokens_in":12731,"feed_emoji":"🗣️","tokens_out":9016,"duration_ms":84617,"temperature":0.7,"pith_summary":"The paper aims to establish that a classifier's training dynamics—specifically its average confidence in the label it predicts—can identify Spanish sentences that are valid in more than one regional variety. This matters because 'common examples' are frequent in Spanish, especially between closely related varieties, and many current variety-identification datasets label each sentence with a single variety, which misclassifies these ambiguous cases and can bias downstream tasks. The method is tested on two domains, newswire and Twitter, and in both it ranks common examples above random. Along the way the paper contributes the first dataset aimed at Cuban and Caribbean Spanish variety identification, built from 1,762 manually annotated tweets.","feed_headline":"Confidence scores expose Spanish sentences valid across varieties","feed_subtitle":"The method beats random ranking on news and Twitter, and a new Cuban Spanish dataset makes Caribbean varieties testable.","key_machinery":"The central object is a modified Datamaps scorer that replaces gold-label confidence with predicted-label confidence: $\\mathrm{DMmean\\text{-}pred} = -\\frac{1}{E}\\sum_{e=1}^{E}\\max_j p_{i,j,e}$, the negative average, over $E$ epochs, of the maximum class probability the model assigns to example $i$. A second score, $\\mathrm{DMstd\\text{-}pred} = \\sqrt{\\frac{1}{E}\\sum_{e=1}^{E}(\\max_j p_{i,j,e} + \\mathrm{DMmean\\text{-}pred})^2}$, tracks epoch-to-epoch variability of that same maximum. The load-bearing change from earlier training-dynamics work is that the metrics are computed on the predicted label rather than the gold label, so the scorer can flag examples the model never confidently settles on—the signature of a sentence that is valid in several varieties. A random scorer serves as the control baseline, and feature-attribution analysis is used to inspect what drives the errors.","core_discovery":"The paper claims that the difficulty a binary variety-identification model has with 'common examples'—sentences valid in more than one Spanish variety—leaves a trace in training dynamics that can be harvested automatically. Instead of scoring each example by confidence on its gold label, as earlier Datamaps work did, the authors score by the average maximum probability assigned to the predicted label across epochs (DMmean-pred), with higher scores meaning lower confidence; a companion score DMstd-pred measures epoch-to-epoch variability. On both the Spanish subset of DSL-TL (Argentina versus Spain) and a new Twitter corpus of Cuban versus non-Cuban Spanish, the confidence-based ranking places common examples ahead of a random ranking, and confidence consistently outperforms variability. The paper also presents its CUBAN SPVARIETY dataset, 1,762 tweets annotated by three native Cuban speakers with labels for Cuban, non-Cuban, and common examples, as the first dataset focused on identifying a Cuban or other Caribbean Spanish variety.","pith_inferences":["The same confidence-ranking signal could serve as an acquisition function for active learning: label the highest-scoring examples first, since they carry the most ambiguity.","A direct test of the mechanism would compare the ranking against human disagreement rates: if DMmean-pred truly tracks commonness, its top-ranked sentences should be the ones annotators most often split on, a pattern the paper's partial-agreement statistics suggest but do not use as the ranking criterion.","In a multi-variety setting beyond binary pairs, training a classifier for each pair of varieties and combining the confidence scores could identify sentences valid across several varieties at once, extending the binary design to the broader annotations the dataset guidelines anticipate."],"forward_implications":["Dataset builders can prioritize re-annotation of the highest-scoring examples instead of checking every sentence by hand.","The scorer can act as an automatic error signal for existing single-label variety datasets, flagging candidates for a 'both' or 'neither' class like the one DSL-TL introduced.","For Cuban Spanish, the new dataset makes it possible to measure how strongly topic words such as Cuba and SOSCuba drive variety predictions, and the error analysis shows those signals need to be controlled.","The appendix's results indicate that multi-class, one-binary-classifier-per-variety models substantially outperform single-label classification on both datasets, reinforcing the need to handle common examples explicitly."],"supporting_citations":[{"why":"Supplies the Datamaps training-dynamics method that this paper adapts by switching confidence tracking from gold labels to predicted labels.","marker":"Swayamdipta et al. (2020)"},{"why":"Provides the DSL-TL Spanish subset with true labels and the third 'common' class, giving the method one of its two test beds.","marker":"Zampieri et al. (2024)"},{"why":"Prior adaptation of training dynamics to human label variation, motivating the predicted-label formulation used here.","marker":"Weber-Genzel et al. (2024)"},{"why":"Provides BETO, the Spanish BERT model whose fine-tuned predicted-label confidence is tracked across epochs.","marker":"Canete et al. (2020)"},{"why":"Supplies the SHAP feature attributions used in the error analysis to show which words push predictions toward one variety.","marker":"Lundberg and Lee (2017)"},{"why":"Establishes the multi-label/common-example problem for dialect identification and supplies the argument that many apparent errors are valid alternative labels.","marker":"Keleg and Magdy (2023)"},{"why":"Shows common examples are the recurring failure class in a VarDial system, motivating automatic detection of these instances.","marker":"Vaidya and Kane (2023)"}],"fun_headline_variants":["Confidence ranking beats variability in Spanish common example detection","New Cuban Spanish dataset targets cross-variety sentences","Model confidence exposes Spanish sentences valid in multiple dialects","Training dynamics identify common examples across Spanish varieties"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a sentence's low confidence in the model's predicted label is driven by its being valid in more than one Spanish variety rather than by unrelated label noise or topic bias, yet in the Cuban dataset common examples were randomly assigned to one of two labels before training, manufacturing exactly such noise.","fun_headline_variants_meta":{"raw":{"variants":["Confidence ranking beats variability in Spanish common example detection","New Cuban Spanish dataset targets cross-variety sentences","Model confidence exposes Spanish sentences valid in multiple dialects","Training dynamics identify common examples across Spanish varieties"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1668,"prompt_tokens":957,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":651}},"tokens_in":573,"tokens_out":711,"duration_ms":7255,"temperature":1.0,"reasoning_tokens":651,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:36:46.717149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the DMmean-pred ranking on a version of CUBAN SPVARIETY where the random binary labels assigned to common examples are flipped; if the precision of the top-500 ranked common examples stays essentially unchanged, the score is tracking the manufactured label noise, not the commonness, and the central claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior adaptation of training dynamics to human label variation, motivating the predicted-label formulation used here."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BETO, the Spanish BERT model whose fine-tuned predicted-label confidence is tracked across epochs."}],"review_version":1}