{"id":"7331a153-7bbd-48dd-a250-d19048fbcd62","arxiv_id":"2506.16322","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A small Polish BERT classifier proved more robust than larger fine-tuned LLMs at classifying safe versus unsafe Polish content, including under character-level adversarial perturbations.","lead":"Researchers built PL-Guard, a human-checked Polish-language dataset for testing whether AI chatbots produce safe answers, plus a noisy version with typos and character swaps to stress-test models. They found that a small, Polish-specific BERT-style classifier was more robust than much larger language models, especially on the noisy test, which matters for deploying safety guards in non-English languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HerBERT's advertised advantage may be an artifact of leakage: PL-Guard-test is carved from the same 7k annotation pool as training with no deduplication, and PL-Guard-adv is a perturbed copy of that same test set.","rationale":"The paper contributes a genuinely useful Polish safety resource, but the central empirical claim is not currently supported. The most direct threat to that claim is the split construction: the test set is a subset of the same 7,000+ annotated instances that form the training set, with no deduplication or independence check. Given template-based prompt generation, near-duplicates are a concrete risk, and the adversarial test set is just a perturbed copy of the same test set, so any leakage inflates both the clean and adversarial HerBERT results. This is the mechanism that could make HerBERT look better than it actually is. The Reader's Appendix C.2 concern is real and independently invalidates the evaluation protocol, but its bias runs in the opposite direction: selecting Llama hyperparameters on the test set can only make the Llama baselines look better, not HerBERT. Thus the Reader's primary mechanism does not explain a spurious HerBERT advantage; the shared-pool/no-dedup issue does. I agree with the REJECT verdict and would keep it, but the most load-bearing reason is the potential train/test leakage, not the hyperparameter selection. A deduplication audit is the decisive check: if HerBERT's advantage survives after removing near-duplicates, the core result may still hold; if not, the headline claim collapses.","tokens_in":13442,"tokens_out":9087,"duration_ms":112373,"concrete_test":"Compute exact and near-duplicate overlap between PL-Guard-train and PL-Guard-test (e.g., normalized Levenshtein distance < 0.1 or embedding cosine > 0.95), report the fraction of test items with a near-duplicate in train, then rerun Table 3 on a deduplicated split or on an independently collected Polish safety test set. If HerBERT's PL-Guard-adv margin over PolyGuard-Qwen and Llama-Guard shrinks or reverses, the central claim is leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—HerBERT achieves the highest overall performance, especially under adversarial conditions—depends on PL-Guard-test being a valid held-out set. Section 3.1 says the 900 test items were selected from the same pool of over 7,000 manually annotated instances, with the remaining 6,487 used for training, and no random split, deduplication, or independence check is reported. Because the harmful prompts were generated with a template-based RainbowTeaming framework, near-duplicate test/train items are plausible. If leakage exists, a fine-tuned BERT classifier can exploit surface memorization, and PL-Guard-adv inherits the same leakage because it is a perturbed copy of exactly those test items. Reported adversarial robustness could then reflect remembering the clean test items rather than genuine robustness. Appendix C.2's statement that the best results on the PL-Guard test set were obtained using certain configurations is a separate validity violation, but its bias runs against HerBERT (it can only make Llama models look better), so it is the shared-pool/no-dedup issue that most directly threatens the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PL-Guard, a Polish-language safety classification benchmark with manually verified labels and an adversarial perturbation variant (PL-Guard-adv). The authors fine-tune three models (HerBERT-based classifier, Llama-Guard-3-8B, and PLLuM, a Polish-adapted Llama) using combinations of PL-Guard, translated WildGuard, and PolyGuard data, and compare them against several baselines including GPT-4 variants and PolyGuard. The central claim is that the HerBERT-based classifier achieves the highest overall performance, particularly under adversarial conditions, despite being much smaller than the Llama-based models.","tokens_in":13662,"tokens_out":3892,"duration_ms":43499,"significance":"If the evaluation were clean, the paper would make a valuable contribution: a manually annotated Polish safety benchmark, an adversarial robustness test set, and evidence that a compact, language-specialized classifier can outperform much larger general-purpose guard models in a medium-resource language. The released resources (test sets and the HerBERT model) would be useful for future research on non-English safety moderation. The high inter-annotator agreement (Krippendorff's alpha = 0.92) and the explicit release of the dataset and model are strengths. However, the validity of the central claim currently hinges on whether the test set is a truly held-out set, which is questionable for the reasons detailed below.","major_comments":[{"comment":"The statement in Appendix C.2 that 'the best results on the PL-Guard test set were obtained using the following configurations' indicates that the PL-Guard test set was used for hyperparameter selection for the Llama-based models. This violates the requirement that a test set remain untouched until final evaluation. While the bias introduced here runs against the paper's favored model (HerBERT was trained with fixed hyperparameters described in Appendix C.1), it nevertheless invalidates the reported Llama-guard and PLLuM results as honest held-out estimates, and it prevents a fair comparison. The authors must either use a separate validation split or clearly report that the test set was used for model selection and adjust the conclusions accordingly.","section":"Appendix C.2 and Section 4.4"},{"comment":"The test set is carved from the same pool of over 7,000 manually annotated instances, with 900 items selected for testing and the remaining 6,487 used for training, and no deduplication or independence check is reported. Because the harmful prompts were generated with a template-based RainbowTeaming framework, the risk of near-duplicate or templatically overlapping content between train and test is substantial. A fine-tuned BERT classifier can exploit such surface-level overlap, and since PL-Guard-adv is a perturbed copy of exactly those test items, the reported adversarial robustness of HerBERT could reflect memorization of the clean test instances rather than genuine robustness. The paper should quantify the duplicate/near-duplicate rate between train and test, and if any significant overlap exists, re-split the data and re-run all experiments.","section":"Section 3.1"},{"comment":"The abstract claims that HerBERT 'achieves the highest overall performance,' but in Table 3, for the binary safety F1 score on the original PL-Guard test set, Llama-Guard-3-8B trained on PL-Guard+WG+PG reaches 0.938 while HerBERT under the same condition reaches 0.935. HerBERT's advantage is clear on the adversarial set and on category-level F1, but 'highest overall performance' is an overstatement unless 'overall' is defined to exclude the binary clean-test comparison. The claim should be narrowed to, for example, the best performance on adversarial inputs and on fine-grained category classification, or the results should be reported with appropriate caveats.","section":"Abstract / Section 5.1"}],"minor_comments":[{"comment":"The learning-rate values '1e7' and '1e5' are almost certainly typos for 1e-7 and 1e-5; this should be corrected to avoid confusion.","section":"Appendix C.2, Table 8"},{"comment":"The WildGuard test subset size is given as 1,709 in Table 1 but as 1,308 in the text of Section 4.3.2; these numbers should be reconciled.","section":"Table 1 and Section 4.3.2"},{"comment":"The paper says in Section 3 that the authors 'plan to release the test portion' of the dataset, but the abstract and conclusion state that the dataset is released. The release status should be consistent.","section":"Section 4.3.3 (not present; see Section 3)"},{"comment":"The examples in Table 2 contain apparent rendering artifacts, such as '♂shield-altsafe' and '/exclamati⌢n-triangle', which make the adversarial examples difficult to read; these should be cleaned up.","section":"Table 2"},{"comment":"There are minor grammatical issues, e.g., 'leaving majority of global languages underexamined' should be 'leaving the majority of global languages underexamined'; a careful copyedit is recommended.","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The two test-set issues (use of the test set for Llama hyperparameter selection and the shared-pool/no-dedup design) are serious but potentially fixable with a re-split and re-evaluation. The shared-pool issue is the more fundamental one because it directly threatens the central adversarial-robustness claim. If the authors can show that no near-duplicates exist, or re-run all experiments on a truly held-out test set, the paper could become a solid contribution. If the contamination turns out to be substantial, the benchmark's validity would be in question and rejection would be warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PL-Guard is a genuinely useful new resource for Polish safety classification, but the headline claim that HerBERT beats the Llama-class models is not established because the test set is not cleanly held out. The paper does two things well: it contributes a manually annotated, Polish-specific safety benchmark (900 balanced test items, 6,487 training items, with high annotator agreement of 0.92), and it adds a typo/OCR-style adversarial variant plus an English translation. Releasing the test set and the fine-tuned HerBERT model is a real service to the community.\n\nThe soft spots are real and central. Section 3.1 says the test items were selected from the same pool of over 7,000 annotated instances, with the remainder used for training; no deduplication or independence check is reported. Because the harmful prompts were generated with a template-based RainbowTeaming framework, near-duplicate surface forms across the split are plausible. If that happened, a fine-tuned BERT can exploit memorization, and since PL-Guard-adv is a perturbed copy of exactly those test items, the adversarial robustness numbers inherit the same contamination. So the main result could be an artifact of split leakage rather than genuine generalization.\n\nAppendix C.2 adds a second, separate problem: the best Llama configurations were chosen on the PL-Guard test set. That makes the Llama numbers optimistic, not pessimistic, so it works against the HerBERT-superiority narrative rather than for it. But it still invalidates the reported comparison as an unbiased estimate. The lack of significance testing and unverified machine translations are minor by comparison; the authors disclose the translation issue themselves.\n\nI would not take the HerBERT conclusion at face value, but I would not dismiss the paper. The dataset is the contribution, and the evaluation protocol is fixable: split with deduplication, select hyperparameters by cross-validation or a separate validation set, then retrain and report. If the corrected protocol still shows a small specialized model holding its own, that is a solid, publishable finding. If it does not, the benchmark itself still deserves a home.","headline":"PL-Guard is a useful new dataset, but the HerBERT-superiority result is not yet credible because the test set may be contaminated by training-split leakage and test-set-tuned hyperparameters.","tokens_in":14185,"tokens_out":2333,"would_cite":true,"duration_ms":24503,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a Polish-language safety benchmark and claims that a small HerBERT classifier outperforms much larger guard models, especially on adversarial noise.","keywords":["Polish-language safety","safety classification","adversarial perturbations","HerBERT","Llama Guard taxonomy","content moderation","multilingual LLM evaluation","benchmark dataset"],"falsifier":"Compare all models on a test set built from chatbot conversations with real Polish users rather than LLM-generated prompts, after removing any near-duplicates with training data; if HerBERT's adversarial margin disappears, the benchmark construction rather than model architecture caused the result.","tokens_in":13258,"feed_emoji":"🛡️","tokens_out":11303,"duration_ms":118052,"temperature":0.7,"pith_summary":"This paper introduces PL-Guard, a manually annotated Polish-language safety benchmark for classifying prompts and model responses as safe or as one of fourteen hazard categories, together with PL-Guard-adv, a version of the test set corrupted by realistic character-level noise such as typos, OCR errors, and diacritic changes. The authors fine-tune three safety models on Polish data, including an 8-billion-parameter Llama-Guard model and a Polish-adapted Llama variant, and compare them with public guard models such as PolyGuard and GPT-4o-mini. Their central claim is that the smallest model, a HerBERT-based classifier, achieves the highest overall performance and is the most robust under adversarial perturbations. This matters because safety evaluation has been heavily English-centric, and if the claim holds, effective Polish safety moderation can be done with a compact, fast, language-specific model rather than a very large general-purpose one.","feed_headline":"Small Polish BERT model tops larger guard models","feed_subtitle":"HerBERT classifier beat Llama-Guard and GPT-4o-mini on Polish safety, especially under typos and character noise.","key_machinery":"The argument rests on two coupled resources. PL-Guard is a manually annotated Polish test set of 900 balanced examples (50 per hazard category plus 200 safe), using the Llama Guard taxonomy of 14 risk categories; PL-Guard-adv is the same set passed through a perturbation pipeline that applies 1 to 20 random character-level edits per sentence (typos, OCR errors, diacritic removal, swaps). The model comparison is carried by HerBERT, a base-sized Polish BERT derivative, fine-tuned as a single-label safety classifier and evaluated against Llama-Guard-3-8B, PLLuM-8B, PolyGuard, WildGuard, and GPT-4 variants. The mechanism that explains the headline result is native-language specialization: the winning classifier sees original Polish text, while the larger models are trained with machine-translated data from other taxonomies. The perturbation pipeline is what exposes the robustness gap.","core_discovery":"The paper's central claim is that a fine-tuned HerBERT-base-cased classifier is the best overall safety moderator for Polish among all tested models. On the clean PL-Guard test set its multiclass category F1 reaches 0.663 and its binary safety F1 reaches 0.935; on the adversarially perturbed PL-Guard-adv set it reaches 0.913 binary F1, the highest of any model, while the best fine-tuned Llama-Guard-3-8B drops to 0.814. The authors explain this by native-language specialization: HerBERT was trained on original Polish text, while the larger models had to rely partly on machine-translated examples, and the perturbed set exposes the larger models' sensitivity to surface noise. They conclude that compact, language-specific classifiers can be both more accurate and more robust than large general-purpose guard models in medium-resource languages.","pith_inferences":["The robustness gap may partly reflect how subword tokenizers handle typos; testing diacritic-only perturbations separately would show whether HerBERT's advantage is a tokenizer property rather than a training-data property.","Because PL-Guard train and test come from a single annotated pool and the appendix reports using the test set to choose Llama hyperparameters, a near-duplicate analysis and a truly held-out re-run would show how much of the reported ranking is benchmark artifact.","A direct inference-cost or latency measurement would make the practical claim about lightweight deployment explicit; the paper argues efficiency but does not report runtime.","The same annotation protocol could be repeated for other medium-resource languages; if the pattern recurs, it would form a design rule: build a small native classifier before renting a large multilingual guard API."],"forward_implications":["If the result holds, Polish-language content moderation can be run on a compact transformer rather than an 8B-parameter model, with lower serving cost and similar or better safety accuracy.","Benchmark designers for non-English languages should include adversarially perturbed splits; clean-set results alone would have ranked Llama-Guard ahead of HerBERT on binary safety.","Adding machine-translated PolyGuard and WildGuard data improves large Llama-based models, but for HerBERT the best binary adversarial safety score comes from the native Polish training set alone.","Polish-trained guard models do not generalize to English, so cross-lingual transfer cannot be assumed and language-specific safety classifiers remain necessary.","For fine-grained hazard classification, the specialized HerBERT model outperforms all tested baselines on Polish, suggesting that taxonomy coverage matters more than raw model scale."],"supporting_citations":[{"why":"Supplies HerBERT, the base Polish BERT model that becomes the paper's best-performing classifier.","marker":"(Mroczkowski et al., 2021)"},{"why":"Defines the 14-category Llama Guard safety taxonomy used for annotation, training, and evaluation, and provides the original Llama Guard baseline.","marker":"(Inan et al., 2023)"},{"why":"Supplies WildGuard/WildGuardMix, the translated training subset and the domain-shift test set used in the experiments.","marker":"(Han et al., 2024)"},{"why":"Supplies PolyGuard, the large multilingual training corpus and the Qwen/Ministral baseline models compared against.","marker":"(Kumar et al., 2025)"},{"why":"Provides the bidirectional translation model used to create Polish WildGuard data and the English version of PL-Guard.","marker":"(Kot et al., 2025)"},{"why":"Supplies the automatic red-teaming framework that generated the initial harmful and benign Polish prompts for the dataset.","marker":"(Krasnodębska et al., 2025)"},{"why":"Motivates the adversarial evaluation by showing that character-level typos can break textual classifiers.","marker":"(Chrabąszcz et al., 2025)"},{"why":"Supplies PLLuM, the Polish-adapted Llama-8B model that is fine-tuned and used to generate answers for the dataset.","marker":"(PLLuM Consortium, 2025)"}],"fun_headline_variants":["HerBERT beats Llama-Guard on Polish safety","Small Polish BERT wins under adversarial noise","Compact HerBERT tops larger guards in Polish test","Language-specific model outperforms general guards","Polish BERT classifier leads robustness test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of models assumes PL-Guard's test set is a clean held-out measure; the appendix reports that the best Llama training configurations were chosen using results on this test set, and the train and test examples come from the same annotated pool with no described deduplication step.","fun_headline_variants_meta":{"raw":{"variants":["HerBERT beats Llama-Guard on Polish safety","Small Polish BERT wins under adversarial noise","Compact HerBERT tops larger guards in Polish test","Language-specific model outperforms general guards","Polish BERT classifier leads robustness test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2647,"prompt_tokens":892,"completion_tokens":1755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1688}},"tokens_in":508,"tokens_out":1755,"duration_ms":15974,"temperature":1.0,"reasoning_tokens":1688,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:43:57.046465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare all models on a test set built from chatbot conversations with real Polish users rather than LLM-generated prompts, after removing any near-duplicates with training data; if HerBERT's adversarial margin disappears, the benchmark construction rather than model architecture caused the result.","supporting_citations":[{"cited_title":"In Pro- ceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12574– 12584","cited_arxiv_id":null,"evidence_quote":"Supplies WildGuard/WildGuardMix, the translated training subset and the domain-shift test set used in the experiments."}],"review_version":1}