{"id":"fa9b6f32-dfc2-47fc-b984-66f94a76c036","arxiv_id":"2506.21508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A nine-task Slovak-language understanding benchmark with translated and newly curated datasets, plus the first broad fine-tuned model comparison for Slovak.","lead":"skLEP is a new benchmark with nine tasks for measuring how well AI language models understand Slovak, built by translating English resources and adding new Slovak datasets. It provides the first broad comparison of Slovak, multilingual, and English models on Slovak, so it gives Slovak NLP a shared yardstick.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validity of the three machine-translated skLEP tasks (RTE, NLI, STS) is the load-bearing premise; the 100-sample relabeling check is too small to rule out label shifts large enough to reorder models.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: machine-translated instances with retained labels may not measure Slovak NLU. My stress-test agrees that this is the central risk. The paper's own evidence is a 100-sample relabeling check and a 90-sentence MT quality evaluation, both conducted by co-authors; these are too thin to guarantee that label noise or translationese does not perturb the benchmark's validity. The reader's verdict of CONDITIONAL is appropriate, and our concern does not change it; it reinforces the condition that the translated tasks be validated more rigorously. I considered whether the unavailability of data and code (promised only after acceptance) is even more load-bearing, but that is a reproducibility issue rather than a threat to the internal validity of the benchmark's construction. The translation concern, by contrast, directly affects whether the benchmark measures what it claims. Therefore, the verdict remains CONDITIONAL, pending stronger validation of the translated tasks.","tokens_in":30743,"tokens_out":6655,"duration_ms":75271,"concrete_test":"Have three independent native Slovak annotators (not co-authors, given the original task guidelines) re-label a stratified random sample of 1,000 instances from each of the NLI, RTE, and STS dev/test sets. Then recompute the mean task score for each of the 14 fine-tuned models using the corrected labels instead of the original ones and compare the resulting ranking of models with the ranking in Table 5. If the Spearman rank correlation of the 14 model scores (averaged over the three tasks) is below 0.9, or if any model's aggregate position changes by more than one place, the benchmark's translated-task validity is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that skLEP measures Slovak natural language understanding. Three of the nine tasks (RTE, NLI, STS) are machine translations of English benchmarks, and the paper's own experiments leave open the possibility that translation errors are more than a minor nuisance. For NLI, the 392,702 training instances were translated with MADLAD-400-3B, the weakest system in the paper's own quality ranking, and were not post-edited. The validating evidence for this choice is a single 100-sample relabeling exercise (Appendix B) that found 2% label-changing errors for RTE, 5% for NLI, and an STS MAE of 0.69 on a 0--5 scale. The sample is small, the annotators are co-authors, and the check covers only training data, not the post-edited test sets. If the full-corpus error rate is higher, or if translationese systematically changes the difficulty distribution, then scores on these tasks may reflect the ability to handle machine-translation artifacts rather than Slovak NLU. Since the paper's contribution is a benchmark, this issue cuts to the core: model rankings and aggregate scores, including the headline comparison of SlovakBERT versus mDeBERTaV3, could shift. The paper is candid about the limitation, but the current evidence is not strong enough to fully secure the claim that all nine tasks are valid Slovak NLU measures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces skLEP, a nine-task benchmark for Slovak natural language understanding, covering token-level (UD, UNER, WGSK), sentence-pair (RTE, NLI, STS), and document-level (HS, SA, QA) tasks. Six tasks are curated from existing Slovak resources or newly created datasets; three tasks (RTE, NLI, STS) are machine-translated from English with native-speaker post-editing of the validation and test sets. The authors evaluate 14 pretrained encoder-only models under a uniform fine-tuning protocol with an extensive hyperparameter search (4,024 runs), report both absolute scores and relative error reduction (RER) against SlovakBERT, and release a toolkit and leaderboard. The paper claims to provide the first comprehensive GLUE-style benchmark for Slovak NLU.","tokens_in":30993,"tokens_out":6233,"duration_ms":65187,"significance":"If the translation-quality concerns are resolved, skLEP would fill a clear gap for Slovak NLP: it provides a standardized evaluation suite, the first systematic comparison of Slovak-specific and multilingual encoder models on Slovak, and a public leaderboard that can drive future research. The work has notable strengths: duplicate removal, explicit native-speaker post-editing for test sets, transparent documentation of the hyperparameter search, and a published translation-quality assessment. The open-source toolkit and detailed appendices support reproducibility. The benchmark's novelty as the first Slovak GLUE-style suite appears justified by the related-work survey.","major_comments":[{"comment":"The validity of the three machine-translated tasks (RTE, NLI, STS) is the load-bearing premise for the claim that skLEP measures Slovak NLU, but the supporting evidence is insufficient. The relabeling check in Appendix B uses only 100 samples per task, with co-author annotators, and the NLI training set was translated by MADLAD-400-3B, the worst-ranked system in the paper's own quality evaluation, without post-editing. The Limitations section admits that these training sets 'have not been manually corrected.' With 5% label-changing errors in NLI, the 392,702 training instances could contain tens of thousands of mislabeled pairs, which may shift fine-tuning outcomes and model rankings. The paper should provide a larger, independent error study (e.g., several hundred samples per task with non-author annotators), report label-error rates on the test sets separately, and analyze the sensitivity of model rankings to the inclusion of the translated tasks (e.g., by reranking models on the six non-translated tasks alone). Without such analysis, the claim that all nine tasks measure Slovak NLU is not fully supported.","section":"Section 3.2, Section 4, Appendix B"},{"comment":"The decision to use MADLAD-400-3B for the NLI training corpus is justified solely by cost, despite this system ranking lowest in the paper's own translation-quality assessment (mean rank 2.54 vs. DeepL's 1.81; fluency 3.48 vs. 3.70). The paper should demonstrate that this choice does not materially affect the NLI task's validity, for example by comparing models fine-tuned on a sample of MADLAD-translated data against the same models fine-tuned on DeepL-translated or post-edited data, or by quantifying the downstream effect of the observed label noise. As it stands, NLI scores may reflect robustness to machine-translation artifacts rather than Slovak inference ability.","section":"Section 4, Tables 3 and 4"}],"minor_comments":[{"comment":"The abstract states that the benchmark data, toolkit, and leaderboard are released, but Section 6 says 'We will release all software and data upon acceptance' and that translated datasets are 'not yet public.' These statements should be aligned; providing anonymized access to the data during review would strengthen the reproducibility claim.","section":"Abstract and Section 6"},{"comment":"The header row of Table 5 is garbled (e.g., 'A VG', 'RERAvg', 'RERF1' run together), making the column structure difficult to parse. The table should be reformatted so each task and metric is clearly separated.","section":"Table 5"},{"comment":"Appendix A mentions 'MobileBERT models on the QA task' as unsupported, but no MobileBERT model appears in the paper's model list or in Table 5. This appears to be a leftover from an earlier draft and should be removed or corrected.","section":"Appendix A"},{"comment":"The post-editing quality experiment uses 30 post-edited and 30 non-post-edited samples, but the post-edited set was 'manually chosen' for inclusion. The statement that 'only 15/60 samples needed post-editing' is therefore not an estimate of the population rate and should be phrased accordingly to avoid overinterpretation.","section":"Appendix C"},{"comment":"The RER values are computed relative to SlovakBERT, a model developed by an overlapping group. The paper should note explicitly that RER is baseline-dependent and that negative RER does not imply below-chance performance; a brief sensitivity check with an alternative baseline would strengthen the interpretation.","section":"Section 5.4"},{"comment":"The WGSK description says the dataset addresses limitations of 'silver-standard resources' but does not specify what those limitations are or how WikiGoldSK's gold-standard quality was verified. A sentence clarifying the annotation procedure and quality checks would improve transparency.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript includes several tasks and the baseline model (UNER, WGSK, QA, SlovakBERT) that originate from or overlap with the authors' own projects, and the translation-quality annotators are co-authors. I do not see evidence of deliberate bias, but the overlap should be disclosed explicitly, e.g., through a competing-interests statement. The novelty claim of being the first comprehensive Slovak NLU benchmark appears justified, but the 'comprehensive' qualifier is currently weakened by the insufficient validation of the machine-translated tasks. The required additional analyses (larger independent error studies and ranking-sensitivity checks) are feasible within the scope of a revision, which is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: skLEP is the first real GLUE-style benchmark for Slovak, and it fills a genuine gap. The authors assembled nine tasks spanning token, sentence-pair, and document level, translated RTE/NLI/STS with native-speaker post-editing, created new splits for HS and QA, and ran the first systematic comparison of 14 encoder models on Slovak. That is real work, and the paper documents it carefully: duplicate removal, label checks, hyperparameter sweeps (over 4,000 runs), and an RER metric that avoids the standard simple-average distortion. The paper is also unusually honest about its limits—it explicitly says the NLI training set was machine-translated with the weakest system and not corrected, and that translationese may persist. That candor counts.\n\nThe soft spots are in proportion. First, the benchmark artifacts are only promised after acceptance, so the core deliverable can't be inspected now. That's a practical issue, not a scientific one, but for a benchmark paper it is central. Second, the validity of the three translated tasks rests on a 100-sample relabeling check per task and a 30-sample post-editing experiment, both with co-authors as annotators. For NLI the label-changing error is about 5%, but with 100 samples the confidence interval is wide, and translationese could shift difficulty or labels enough to reorder models. The paper acknowledges this in the Limitations, but the evidence is thinner than the claim 'benchmark measures Slovak NLU' would like. Third, ModernBERT is not evaluated on QA, yet its average is compared to other models that include QA; the paper mentions this only in passing, and the headline table makes it easy to miss. Fourth, no per-seed variance or human baseline is reported. Mild self-citation (several tasks and the RER baseline come from overlapping groups) is worth remembering but does not change the results.\n\nOverall, this is a good and honest contribution for Slovak NLP and for the GLUE-for-every-language family. The central design is sound; the translation-quality question is real but not disqualifying, and the authors already did more checking than most benchmark papers do. It should go to review, with the expectation that the release lands and the translation-validation evidence gets stronger. I'd cite it once the data is public.","headline":"A solid, honest first Slovak GLUE-style benchmark, with one load-bearing translation-quality question that needs more evidence before the rankings are taken as settled.","tokens_in":31639,"tokens_out":4439,"would_cite":true,"duration_ms":45366,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims skLEP is the first GLUE-style benchmark for Slovak natural language understanding, with nine tasks at token, sentence-pair, and document levels, and a first fine-tuned comparison of fourteen pretrained models.","keywords":["Slovak natural language understanding","GLUE-style benchmark","pretrained language model evaluation","machine translation pipeline","Relative Error Reduction","token-level and sentence-pair tasks","mid-resource language NLP","leaderboard"],"falsifier":"Re-run the paper's Appendix B relabeling experiment at scale: have native Slovak annotators independently re-label a few thousand random instances from the RTE, NLI, and STS training and test sets, then compare model rankings computed on the corrected labels against the published ones. If the label-change rate stays in the single digits and the rankings are unchanged, the translation premise holds; if corrected labels move which model leads or meaningfully reorder the table, the benchmark's conclusions on the translated tasks would need revision.","tokens_in":30483,"feed_emoji":"🇸🇰","tokens_out":8656,"duration_ms":80399,"temperature":0.7,"pith_summary":"Slovak, a mid-resource language with about ten million speakers, has had no standardized suite for measuring natural language understanding, which makes it hard to compare the several Slovak-specific language models that have recently appeared. The paper claims to fill that gap with skLEP, a GLUE-style benchmark of nine tasks covering token-level, sentence-pair, and document-level understanding, built partly from original Slovak datasets and partly from English tasks translated with native-speaker post-editing. It then reports the first systematic fine-tuned comparison of fourteen Slovak, multilingual, and English pretrained models on these tasks, using relative error reduction against SlovakBERT as the headline metric. The result a sympathetic reader would take away: the multilingual mDeBERTaV3Base is the current leader, but the older SlovakBERT still beats most newer alternatives, and the hardest open targets are question answering and textual entailment.","feed_headline":"First Slovak NLU benchmark puts nine tasks to the test","feed_subtitle":"Fine-tuned comparison of 14 models: a multilingual DeBERTa leads while SlovakBERT stays competitive.","key_machinery":"The central object is the benchmark itself: nine Slovak NLU tasks with fixed splits and task-specific metrics (Macro F1, accuracy, Pearson correlation), assembled by mixing curated original datasets with machine-translated English tasks. Two mechanisms carry the argument. The first is the translation pipeline, which uses DeepL for most datasets and the open-weights MADLAD-400-3B for the much larger NLI corpus, validated by a five-system translation quality study and by native-speaker post-editing of the development and test sets. The second is the aggregation metric, Relative Error Reduction (RER), taken from the DUMB benchmark, which normalizes each task score by the baseline error rate so that gains on near-solved tasks like UD count more than equal gains on harder tasks like QA; this is what lets the paper rank models on balanced performance rather than on raw averages.","core_discovery":"The paper's central claim is that skLEP is the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding. It comprises nine tasks: part-of-speech tagging, two named-entity-recognition datasets (Universal NER's Slovak subset and the hand-annotated WikiGoldSK), textual entailment, natural language inference, semantic textual similarity, hate speech classification, sentiment analysis, and the SK-QuAD question answering dataset. Three of the nine (RTE, NLI, STS) were produced by machine-translating established English resources and post-editing the development and test splits with native speakers, while the rest are curated original Slovak datasets. On top of the benchmark, the paper presents the first systematic evaluation of fourteen pretrained models fine-tuned on all tasks, reporting both absolute scores and Relative Error Reduction with SlovakBERT as baseline; mDeBERTaV3Base achieves the highest mean score and a 6.43 percentage-point error reduction, and only two models beat SlovakBERT's average. The authors also release the data, a fine-tuning and evaluation toolkit, and a public leaderboard.","pith_inferences":["The paper's pipeline, translate, post-edit, spot-check label stability, is a reusable recipe for bootstrapping benchmarks in other mid-resource languages, and reporting translation-error rates the way this paper does could become a norm for translated benchmarks.","Because the NLI training set was translated with the weaker open-weights model while its development and test sets were post-edited, models fine-tuned on it may internalize translationese patterns; a natively collected Slovak NLI corpus could shift rankings on that task, a testable prediction the paper does not make.","The RER aggregation embeds a value judgment, rewarding balanced competence over peak performance on easy tasks; if other benchmarks adopt it, it changes which models win, so the choice deserves explicit discussion wherever benchmarks are compared.","The leaderboard's policy of fully public test sets trades away the protection that keeps comparable English benchmarks uncontaminated; as skLEP gains users, leakage of test items into pretraining data may become a concern."],"forward_implications":["Slovak NLP gains a public, standardized evaluation suite with a leaderboard, so future models can be compared on identical ground instead of ad-hoc task choices.","The baselines fix a concrete state of the art: mDeBERTaV3Base leads with 6.43 percentage points of RER over SlovakBERT, and any new Slovak model can be measured against this.","SlovakBERT's strong showing suggests language-specific pretraining still pays off for mid-resource languages, even against much newer multilingual and English models.","Question answering (F1 mostly below 75) and textual entailment (accuracy mostly below 70) are the clearest open problems the benchmark exposes.","The reported translation-error rates (2 to 5 percent label changes, STS mean absolute error 0.69) give users a quantitative bound on the noise in the three translated tasks."],"supporting_citations":[{"why":"Provides the GLUE benchmark design that skLEP explicitly follows, including the multi-task structure and task typology.","marker":"Wang, 2018"},{"why":"XNLI is the source of the natural language inference task, translated and post-edited for Slovak.","marker":"Conneau et al., 2018"},{"why":"Supplies the semantic textual similarity task and its averaged similarity scoring methodology.","marker":"Cer et al., 2017"},{"why":"Introduces SlovakBERT, the baseline model used for Relative Error Reduction and the closest prior work on Slovak evaluation.","marker":"Pikuliak et al., 2021"},{"why":"Supplies the Relative Error Reduction aggregation method that the paper uses for its headline model rankings.","marker":"de Vries et al., 2023"},{"why":"Provides SK-QuAD, the manually annotated Slovak question answering dataset used as the QA task.","marker":"Hládek et al., 2023"},{"why":"Provides WikiGoldSK, the hand-annotated Slovak named entity recognition dataset used as the WGSK task.","marker":"Suba et al., 2023"},{"why":"Provides Universal NER, the source of the UNER task's cross-lingually consistent annotations.","marker":"Mayhew et al., 2024"}],"fun_headline_variants":["First Slovak NLU benchmark: nine tasks, 14 models","skLEP: first comprehensive Slovak NLU benchmark","Slovak NLU benchmark: mDeBERTaV3Base beats SlovakBERT","Nine tasks, 14 models: Slovak NLU benchmark debuts","mDeBERTaV3Base leads new Slovak NLU benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that machine-translated English sentences with their original labels measure Slovak natural language understanding rather than translationese; the paper itself concedes that the RTE, NLI, and STS training sets were not manually corrected, and its own 100-sample checks found 2 to 5 percent of labels changed by translation errors, so if those errors shift difficulty or labels systematically across the full corpora, model rankings on these three tasks could change.","fun_headline_variants_meta":{"raw":{"variants":["First Slovak NLU benchmark: nine tasks, 14 models","skLEP: first comprehensive Slovak NLU benchmark","Slovak NLU benchmark: mDeBERTaV3Base beats SlovakBERT","Nine tasks, 14 models: Slovak NLU benchmark debuts","mDeBERTaV3Base leads new Slovak NLU benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000523,"raw_usage":{"total_tokens":2511,"prompt_tokens":907,"completion_tokens":1604,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":1513}},"tokens_in":523,"tokens_out":1604,"duration_ms":14908,"temperature":1.0,"reasoning_tokens":1513,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:23:38.295965+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's Appendix B relabeling experiment at scale: have native Slovak annotators independently re-label a few thousand random instances from the RTE, NLI, and STS training and test sets, then compare model rankings computed on the corrected labels against the published ones. If the label-change rate stays in the single digits and the rankings are unchanged, the translation premise holds; if corrected labels move which model leads or meaningfully reorder the table, the benchmark's conclusions on the translated tasks would need revision.","supporting_citations":[{"cited_title":"SlovakBERT: Slovak Masked Language Model","cited_arxiv_id":"2109.15254","evidence_quote":"Introduces SlovakBERT, the baseline model used for Relative Error Reduction and the closest prior work on Slovak evaluation."},{"cited_title":"DUMB: A Benchmark for Smart Evaluation of Dutch Models","cited_arxiv_id":"2305.13026","evidence_quote":"Supplies the Relative Error Reduction aggregation method that the paper uses for its headline model rankings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides WikiGoldSK, the hand-annotated Slovak named entity recognition dataset used as the WGSK task."}],"review_version":1}