Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that a small proxy model's word-importance scores can steer cheap character- and word-level perturbations that flip large multilingual LLMs' predictions on Polish classification tasks at rates up to 49 percent.

desk verdict A useful Polish robustness benchmark whose central proxy-transfer claim is unproven for lack of LLM random-word baselines. read the letter →

arxiv 2506.07645 v1 pith:AVUXRUKW submitted 2025-06-09 cs.CL

Maciej Chrabąszcz , Katarzyna Lorenc , Karolina Seweryn This is my paper
classification cs.CL
keywords LLMrobustnessadversarialperturbationproxymodelwordimportanceSHAPattributionPolishNLPlow-resourcelanguagesattacksuccessrate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large multilingual language models are surprisingly fragile to tiny, human-plausible perturbations in a low-resource language such as Polish, and that you do not need the target LLM itself to locate the vulnerable words. The authors train a small Polish classifier (RoBERTa), use SHAP attributions to rank words by importance, and then perturb only the top words with simple typos, OCR errors, diacritic drops, and related edits. On the KLEJ Polish classification tasks, these attacks flip the predictions of Bielik, Mistral-7B, and Llama-3.1-8B at rates up to nearly half of all examples (for instance, OCR attacks on the cyberbullying set reach an attack success rate of 0.488 for Bielik). The point is not merely that certain perturbations exist, but that they are cheap to construct and do not require white-box access to the target model, which matters for safety evaluation of multilingual models whose safety training is dominated by English data.

What carries the argument

The load-bearing mechanism is the combination of a small proxy model with an attribution method, here SHAP, to produce a word-importance ranking according to Equation (2), where each word's importance is the mean attribution over its subtokens. The paper adopts the premise that a well-performing small model is a good proxy: Section 3.1 states that if the model performs well on a dataset, it can be considered a good proxy for calculating word importance. That ranking then selects which words to perturb under ten character- and word-level perturbation families (keyboard, OCR, random insertion, deletion, substitution, swapping, diacritics, splits, orthographic errors, and Słowosieć-based relations). ASR in Equation (3) measures the fraction of correctly labelled examples whose prediction changes after perturbation. The framework's cheapness comes from computing attributions on a small classifier rather than on the target LLM.

What would settle it

Run the same perturbation families on the same LLMs with words chosen at random instead of by proxy importance, matched for the number of changed words; if random-word ASR is statistically indistinguishable from proxy-targeted ASR on Bielik, Mistral, or Llama, then the attribution ranking is not doing the work. A complementary check is to compute SHAP importance directly on the target LLM (with a reduced VRAM budget) and compare the top-word sets with the proxy's; if the overlap is at chance, the transfer premise fails.

Watch

Extended reading notes

Core claim

The central discovery is that importance-targeted perturbations are dramatically more effective than random perturbations at low budgets, and that a small proxy model can provide the needed importance ranking for a language the LLM was not specifically safety-tuned on. Concretely, using SHAP on a fine-tuned Polish RoBERTa to select the most important words, then altering a few characters in those words, yields attack success rates of 0.334–0.488 for OCR-style edits on Bielik and Llama across several datasets, with Mistral somewhat lower but still non-trivial. The same framework extends to word-level edits, such as relations from Słowosieć, orthographic errors, and random splits, while diacritic and orthographic errors are the least effective. The authors interpret this as evidence that LLM robustness is not only an English problem and can be probed with modest computational resources.

Load-bearing premise

The load-bearing premise is that word-importance scores computed on a small Polish proxy model transfer to the target LLM, so that the words most important to the proxy are also the words whose perturbation most changes the LLM's prediction.

Editorial extensions

If this is right

  • Developers of Polish-capable LLMs can use the released perturbed datasets as pre-deployment robustness checks, with a threshold such as the 5% ASR cutoff flagging the perturbation families that most need attention.
  • Character-level edits (OCR, insertion, substitution, swapping, deletion) are the most effective at fooling LLMs, so alignment and fine-tuning pipelines for low-resource languages should include such noise in their training data.
  • The framework is language-adaptable: only diacritic dictionaries and word-relation networks need to be swapped for a new language, so the same methodology can probe robustness in other low-resource languages.
  • If simple classification predictions can be flipped this easily, the paper concludes that safety-critical generative outputs could also be at risk from the same perturbations, making this an evaluation concern rather than just an accuracy concern.
  • SHAP-based targeted attacks clearly outperform gradient-based attribution methods like Vanilla Gradient and SmoothGrad, which perform no better than random word selection in the paper's experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported attack success rates are measured on classification tasks, and the paper's inference that the same perturbations would circumvent generative safety mechanisms is plausible but untested; a direct follow-up would perturb key tokens in harmful Polish prompts and measure refusal rates.
  • Because the proxy model is trained on each task's training split, its importance ranking will inherit any biases or weaknesses of that small model; a task where the proxy performs poorly would likely yield a less reliable ranking for the LLM.
  • If the transfer of importance rankings holds across languages, an adversary with a small labelled dataset could automate this pipeline for other low-resource languages, turning the reported fragility into a practical deployment risk.
  • The 5% ASR robustness cutoff is an arbitrary threshold; the operational meaning of 'non-robust' depends on the harm of a single failure, so the cutoff should be calibrated to the specific application being protected.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for evaluating the robustness of large language models (LLMs) on Polish by generating character- and word-level perturbations guided by word-importance scores computed with a small proxy classifier. The authors fine-tune PolBERT, HerBERT, and Polish RoBERTa on five KLEJ datasets, use SHAP (and other attribution methods) to rank words, perturb the top-ranked words with ten perturbation types, and measure attack success rate (ASR) on Bielik, Mistral-7B, and Llama-3.1-8B. They report that targeted perturbations, especially OCR and keyboard errors, achieve high ASR on some datasets and claim this reveals vulnerabilities that could potentially circumvent LLM safety mechanisms. Datasets and code are released.

Significance. If the central claim is supported, the paper would provide a useful Polish robustness benchmark, a systematic perturbation taxonomy that includes language-specific error types, and a cost-saving proxy-model approach to word-importance-based attacks. The paper ships reproducible perturbation recipes and extensive ASR tables that are internally consistent, and the evaluation design avoids circularity because ASR is measured on held-out test sets with target LLMs external to proxy training. The most valuable contribution would be the proxy-transfer result: demonstrating that importance rankings from a small classifier yield stronger attacks on much larger LLMs than random-word selection. That specific result is currently not established, so the significance of the framework rests on an unvalidated assumption.

major comments (4)
  1. [Section 5.3 and Figures 3, 5, 6; Table 4] The central claim that proxy-computed word importance enables 'surprisingly strong attacks' on LLMs is not supported because the target-LLM experiments lack a random-word baseline. For the proxy models, Figures 2 and 4 show dotted random-word curves and demonstrate that SHAP-based selection outperforms random selection. For the LLMs, however, the dotted lines in Figures 3, 5, and 6 represent the fixed 0.05 robustness threshold, not a random-word baseline. Without perturbing the same number of randomly selected words on Bielik, Mistral, and Llama, the reader cannot determine whether the reported ASR values in Table 4 reflect targeted word importance or merely generic corruption from character/word edits. This missing control is load-bearing for the abstract's claim that the proxy model is what makes the attacks cheap and effective.
  2. [Section 3.1 and Section 7] The proxy-to-LLM transfer assumption is asserted but not validated. Section 3.1 states that a model performing well on a dataset 'can be considered a good proxy for calculating word importance,' but no evidence links proxy accuracy to ranking transfer across model families. Section 7 explicitly concedes that 'Importance rankings derived directly from the target LLMs could potentially differ and offer a more precise basis.' The paper needs a concrete transfer test, for example comparing ASR from proxy-selected words against ASR from random words on the LLMs, or comparing proxy rankings with target-LLM rankings on a subset. As written, the contribution of the proxy component is unestablished.
  3. [Abstract and Sections 5.1, 6] The claim that the observed perturbations could be used to 'circumvent [LLMs'] internal safety mechanisms' is unsupported by the experimental design. All evaluations are classification tasks on KLEJ datasets with ASR defined as a change in predicted class. No generative safety task, jailbreak prompt, or test for harmful content generation is conducted, so the paper does not demonstrate safety-mechanism circumvention. This overreach should either be removed or supported by direct safety-oriented experiments; otherwise the conclusion in Section 6 that perturbations 'could be used to deceive models into generating harmful content' is speculation.
  4. [Tables 4 and 8-13; Figure 2] The quantitative comparisons lack statistical uncertainty measures. Each ASR value is a single point estimate with no confidence intervals, repeated runs, or significance tests. This matters for claims such as 'SHAP gives the highest ASR compared to other methods' (Section 5.2), where differences among attribution methods at small word counts are often only a few percentage points, and for comparing perturbation types in Table 4 across datasets of very different sizes (e.g., 494 test examples for P-O). Bootstrapped confidence intervals or multiple evaluation runs are needed before ranking methods or perturbations.
minor comments (5)
  1. [Appendix A] The sentence 'Table 5 contains LLMs performance on original test sets' is inconsistent with the table content: Table 5 shows dataset split sizes, while LLM accuracy and F1 are in Table 6.
  2. [Table 4 and Reference [29]] Table 4 labels the model 'Bielik v1' while the cited technical report describes Bielik 7B v0.1; please clarify which checkpoint was used.
  3. [Table 4 caption] The caption refers to 'smaller models' although the table reports results for 7B and 8B LLMs; 'target LLMs' or 'evaluated LLMs' would be clearer.
  4. [Section 3.3] The sentence 'We believe many language-specific perturbations will not work for Polish' appears to say the opposite of the intended meaning; it should probably read 'will not work for other languages' or be rephrased.
  5. [Figures 5 and 6] Figures 5 and 6 appear only in Appendix C and are not referenced in the main text; consider either referencing them in Section 5.3 or moving them to the main body if they are meant to support the LLM robustness discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all ASR values are measured on external LLMs with held-out test sets, and no fitted parameter is renamed as a prediction.

full rationale

The paper's claimed derivation chain is: train a small proxy model, compute word importance with SHAP, perturb the most important words, then measure ASR on Bielik, Mistral-7B, and Llama-3.1-8B. Every reported ASR is computed with Equation (3) on held-out test sets for target LLMs that are external to the proxy training and to the importance ranking. No parameter is fitted to the LLM ASR values, and no prediction is defined in terms of the result it is said to predict. The proxy-transfer assumption in Section 3.1 ('If the model performs well on a dataset, it can be considered a good proxy for calculating word importance') is an empirical claim about transferability, not a definitional equivalence; the paper explicitly concedes in Section 7 that 'Importance rankings derived directly from the target LLMs could potentially differ,' which is a limitation and a correctness risk, not a circular step. The absence of random-word baselines for the target LLMs weakens support for the proxy's contribution, but that is an experimental validation gap rather than a reduction of the central claim to its own inputs. No self-citation chain is load-bearing, and no external result is repackaged as a derivation. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on two structural assumptions: that proxy-model attribution transfers to LLMs and that attribution scores reflect semantic importance. No invented entities are introduced. The threshold and perturbation budgets are hand-selected rather than derived.

free parameters (3)
  • robustness_threshold_0.05 = 0.05
    Hand-set cutoff for declaring a model non-robust in Figure 3 and Table 4; no derivation or calibration is given.
  • character_perturbation_budget = 1 to min(len(word)*0.15, 4) characters
    Hand-designed budget intended to keep perturbed words human-readable; directly controls attack strength (Appendix B.1).
  • words_changed_range = 1 to 10
    The evaluation sweeps 1 to 10 perturbed words; ASR tables aggregate over this range, so conclusions depend on this choice.
assumptions (3)
  • domain assumption Word importance scores computed with a small proxy model transfer to target LLMs.
    Section 3.1 states a well-performing proxy is good enough to compute importance; the authors never test proxy importance against random selection on the actual LLMs. Section 7 acknowledges rankings could differ.
  • domain assumption Attribution methods such as SHAP faithfully reflect semantic word importance.
    Sections 2.2 and 7 note attribution methods may be unfaithful; the framework depends on this being true for the perturbation ranking.
  • domain assumption ASR conditional on originally correct predictions is a valid robustness measure.
    Equation 3 defines ASR only for cases where the clean model was right; this choice excludes the large fraction of originally wrong LLM predictions shown in Table 6, affecting how the reported vulnerability rates should be interpreted.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models." pith.science (2026). https://pith.science/paper/AVUXRUKW

@misc{pith2026250607645,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVUXRUKW}},
  note         = {Machine review of arXiv:2506.07645}
}
read the original abstract

Large language models (LLMs) have demonstrated impressive capabilities across various natural language processing (NLP) tasks in recent years. However, their susceptibility to jailbreaks and perturbations necessitates additional evaluations. Many LLMs are multilingual, but safety-related training data contains mainly high-resource languages like English. This can leave them vulnerable to perturbations in low-resource languages such as Polish. We show how surprisingly strong attacks can be cheaply created by altering just a few characters and using a small proxy model for word importance calculation. We find that these character and word-level attacks drastically alter the predictions of different LLMs, suggesting a potential vulnerability that can be used to circumvent their internal safety mechanisms. We validate our attack construction methodology on Polish, a low-resource language, and find potential vulnerabilities of LLMs in this language. Additionally, we show how it can be extended to other languages. We release the created datasets and code for further research.

Figures

Figures reproduced from arXiv: 2506.07645 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. We can calculate word importance using a trained proxy model on [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Relation of ASR on proxy models and number of perturbed words for different attribution methods. The [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Relation of ASR on Mistral and the number of perturbed words for different attribution methods. The dotted [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Relation of ASR on smaller LMs and number of perturbed words for different attribution methods. The dotted [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Relation of ASR on Llama and the number of perturbed words for different attribution methods. The dotted [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Relation of ASR on Bielik and the number of perturbed words for different attribution methods. The dotted [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PL-Guard: Benchmarking Language Model Safety for Polish

    cs.CL 2025-06 reject novelty 6.0 of 10

    A small Polish BERT classifier proved more robust than larger fine-tuned LLMs at classifying safe versus unsafe Polish content, including under character-level adversarial perturbations.

Reference graph

Works this paper leans on

40 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu...

    work page 2019

  2. [2]

    Abhishek Kadian

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and et al. Abhishek Kadian. The llama 3 herd of models, 2024

    work page 2024

  3. [3]

    Is bert really robust? a strong baseline for natural language attack on text classification and entailment

    Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. InAAAI Conference on Artificial Intelligence, 2019

    work page 2019

  4. [4]

    Xue, and Xipeng Qiu

    Linyang Li, Ruotian Ma, Qipeng Guo, X. Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert.ArXiv, abs/2004.09984, 2020

    arXiv 2004

  5. [5]

    T3: Tree-autoencoder constrained adversarial text generation for targeted attack

    Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. T3: Tree-autoencoder constrained adversarial text generation for targeted attack. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6134–6150, Online, November 2020. A...

    work page 2020

  6. [6]

    Word-level textual adversarial attacking as combinatorial optimization

    Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors,Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066–6080, Online, July 20...

    work page 2020

  7. [7]

    Jailbroken: How does llm safety training fail? In A

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 80079–80110. Curran Associates, Inc., 2023. 10 Evaluating LLMs Robustness in Less Resourced Languages with Pr...

    work page

  8. [8]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    arXiv 2024

Show all 40 references
  1. [9]

    Unsupervised cross-lingual representation learning at scale

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. InAnnual Meeting of the Association for Comput...

    2019

  2. [10]

    Command r +, 2024

    CohereForAI. Command r +, 2024. Accessed: 2024-08-29

    2024

  3. [11]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  4. [12]

    Textbugger: Generating adversarial text against real-world applications

    Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against real-world applications. InProceedings 2019 Network and Distributed System Security Symposium. Internet Society, 2019

    2019

  5. [13]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023

    2023

  6. [14]

    Adversarial glue: A multi-task benchmark for robustness evaluation of language models

    Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. InAdvances in Neural Information Processing Systems, 2021

    2021

  7. [15]

    Decodingtrust: A comprehensive assessment of trustworthiness in gpt models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. InThirty-seventh Conference on Neural Information Processing ...

    2023

  8. [16]

    Deep inside convolutional networks: visualising image classification models and saliency maps

    K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. InProceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014

    2014

  9. [17]

    A unified approach to interpreting model predictions.Advances in neural information processing systems, 30:4765–4774, 2017

    M Scott, Lee Su-In, et al. A unified approach to interpreting model predictions.Advances in neural information processing systems, 30:4765–4774, 2017

    2017

  10. [18]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier.Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

    2016

  11. [19]

    Not just a black box: Learning important features through propagating activation differences.ArXiv, abs/1605.01713, 2016

    Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences.ArXiv, abs/1605.01713, 2016

    2016 arXiv

  12. [20]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. InInternational Conference on Machine Learning, 2017

    2017

  13. [21]

    Viégas, and Martin Wattenberg

    Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise.ArXiv, abs/1706.03825, 2017

    2017 arXiv

  14. [22]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. InAnnual Meeting of the Association for Computational Linguistics, 2020

    2020

  15. [23]

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 387–396, 2021

    2021

  16. [24]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782–791, June 2021

    2021

  17. [25]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Je...

    2020

  18. [26]

    HerBERT: Efficiently pretrained transformer-based language model for Polish

    Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, and Ireneusz Gawlik. HerBERT: Efficiently pretrained transformer-based language model for Polish. InProceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 1–10, Kiyv, Ukraine, April 2021. Associatio...

    2021

  19. [27]

    Polbert: Attacking polish nlp tasks with transformers

    Dariusz Kłeczek. Polbert: Attacking polish nlp tasks with transformers. In Maciej Ogrodniczuk and Łukasz Kobyli´nski, editors,Proceedings of the PolEval 2020 Workshop. Institute of Computer Science, Polish Academy of Sciences, 2020

    2020

  20. [28]

    Assessing generalization capability of text ranking models in polish,

    Sławomir Dadas and Małgorzata Gr˛ ebowiec. Assessing generalization capability of text ranking models in polish,

  21. [29]

    Bielik 7b v0

    Krzysztof Ociepa, Łukasz Flis, Krzysztof Wróbel, Adrian Gwo´ zdziej, and Remigiusz Kinas. Bielik 7b v0. 1: A polish language model–development, insights, and evaluation.arXiv preprint arXiv:2410.18565, 2024

    2024

  22. [30]

    Openchat: Advancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235, 2023

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data.arXiv preprint arXiv:2309.11235, 2023

    2023 arXiv

  23. [31]

    Promptbench: Towards evaluating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528, 2023

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528, 2023

    2023 arXiv

  24. [32]

    On the robustness of chatgpt: An adversarial and out-of-distribution perspective.arXiv preprint arXiv:2302.12095, 2023

    Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, et al. On the robustness of chatgpt: An adversarial and out-of-distribution perspective.arXiv preprint arXiv:2302.12095, 2023

    2023 arXiv

  25. [33]

    Maziarz, Maciej Piasecki, and Ewa K

    Marek M. Maziarz, Maciej Piasecki, and Ewa K. Rudnicka. Słowosie ´c :polski wordnet : proces tworzenia tezaurusa.Polonica, 34:79–98, 2014

    2014

  26. [34]

    Klej: Comprehensive benchmark for polish language understanding.arXiv preprint arXiv:2005.00630, 2020

    Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. Klej: Comprehensive benchmark for polish language understanding.arXiv preprint arXiv:2005.00630, 2020

    2005 arXiv

  27. [35]

    This is the way: designing and compiling lepiszcze, a comprehensive nlp benchmark for polish

    Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Arkadiusz Janz, Piotr Szyma´nski, Marcin W ˛ atroba, Mikoł aj Morzy, Tomasz Kajdanowicz, and Maciej Piasecki. This is the way: designing and compiling lepiszcze, a comprehensive...

    2022

  28. [36]

    Michal Ptaszynski, Agata Pieciukiewicz, and Paweł Dybała. Results of the poleval 2019 shared task 6: First dataset and open shared task for automatic cyberbullying detection in polish twitter.Proceedings of the PolEval 2019 Workshop, page 89, 2019

    2019

  29. [37]

    Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain consumer reviews

    Jan Koco ´n, Piotr Miłkowski, and Monika Za ´sko-Zieli´nska. Multi-level sentiment analysis of PolEmo 2.0: Extended corpus of multi-domain consumer reviews. InProceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 980–991, Hong Kong, China...

    2019

  30. [38]

    Mistral 7b.arXiv preprint arXiv:2310.06825, 2023

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b.arXiv preprint arXiv:2310.06825, 2023. 12 Evaluating LLMs Robustness in Less...

    arXiv

  31. [2019]

    Association for Computational Linguistics

  32. [2024]

    arXiv:2402.14318 [cs.CL]

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.