Pith. sign in

REVIEW 3 major objections 5 minor 81 references

BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BavGround shows open-weight LLMs are consistently weaker on Bavarian dialect and regional knowledge than on German or English.

desk verdict A genuinely useful regional-culture benchmark with honest limitations, but the paper's headline Bavarian difficulty claim does not survive its own protocol-sensitivity critique. read the letter →

arxiv 2608.12894 v1 pith:ZQR7EKGW submitted 2026-08-13 cs.CL

Jophin John , Michael Hoffmann , Jan Fillies , Michael A. Hedderich , Barbara Plank This is my paper · ORCID
classification cs.CL
keywords BavariandialectculturalgroundingLLMevaluationmultiple-choicebenchmarkprotocolsensitivityregionalculturecontinuedpretrainingmultilingual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces BavGround, a multiple-choice benchmark of 206 questions about Bavarian culture, history, language, and daily life, each translated into English, German, and Bavarian, yielding 618 evaluation instances. Its central claim is that current open-weight instruction-tuned models are systematically worse on Bavarian items and on source-grounded regional questions than on general cultural facts: averaged over fifteen models, Bavarian accuracy is 45.9% versus 57.6% for German, and grounded questions fall to 46.7% versus 63.1% for general-knowledge questions. The paper further claims that the scoring protocol strongly affects measured competence, since switching from answer-letter scoring to length-normalized option-text scoring raises the open-weight mean from 53.0% to 59.3% and changes model rankings. The broader point is that regional cultural competence is not a single capability, and a single multiple-choice score can hide how much of a model's difficulty comes from dialect, local evidence, or answer-format behavior.

What carries the argument

The carrying mechanism is BavGround itself: 206 source questions across eight cultural domains, split into general-knowledge (GEN) and source-grounded (GRD) items, manually translated into English, German, and Bavarian to form 618 parallel instances. Each item can be scored through several protocols—answer-letter log-probability, shuffled labels, length-normalized option-text likelihood, deterministic generation parsed back to labels, semantic embedding of generated answers, and hidden-state alignment—so the benchmark separates answer-content knowledge from label priors, option order, and format behavior. A third component is the GENBA-10B checkpoint series, 85 checkpoints from one continued-pretraining run, which lets the authors observe how cultural knowledge changes during training rather than treating it as a static property of a final model.

What would settle it

Recruit native Bavarian speakers from at least three dialect sub-regions and have them answer the 206 Bavarian items without access to the source documents; if their accuracy is near the open-weight mean of 45.9% or their agreement with the gold answers is low, BavGround is not measuring common Bavarian competence. A second check is to measure the Levenshtein distance and perceived naturalness of the Bavarian translations: if native raters judge them as near-Standard German, the dialect-gap claim loses its target.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that BavGround is a usable, protocol-sensitive instrument for measuring regional cultural grounding and dialect competence, and that current 7B–10B open-weight models have a real and consistent gap on it. Strong multilingual models cluster around 69% under standard letter scoring, but the open-weight mean is 53.0%, with Bavarian items consistently below English and German by 9.8–11.7 percentage points and grounded questions below general-knowledge items by 16.4 points. The same pattern appears in a closed-model reference, which scores 89.6% overall but still drops from 95.0% on general questions to 86.2% on grounded questions. A continued-pretraining analysis of GENBA-10B shows the same unevenness: option-text likelihood improves from 25.4% to 50.8% across checkpoints, while letter accuracy stays low and Bavarian and dialect items remain the weakest throughout. The authors' intended conclusion is that localized cultural evaluation needs to be both domain-aware and protocol-aware, because different scoring views reveal different failure modes.

Load-bearing premise

The load-bearing premise is that the Bavarian items genuinely represent Bavarian dialect and regional usage; the paper reports that all Bavarian translations came from a single native Chiemgau speaker who has lived outside Bavaria for years, so if those items are closer to Standard German than true Bavarian, the measured Bavarian gap understates the real dialect gap.

Editorial extensions

If this is right

  • If the Bavarian gap is real, models deployed for Bavarian-speaking users need dialect-focused data and evaluation, not just more German-language exposure.
  • A single multiple-choice letter score is insufficient for cultural benchmarks, because changing the scoring protocol changes both absolute scores and model rankings.
  • General cultural knowledge does not transfer cleanly to source-grounded regional knowledge, so high performance on broadly accessible facts does not predict performance on regionally specific items.
  • Continued pretraining can improve answer-content likelihood while leaving dialect competence comparatively weak, meaning checkpoint diagnostics should separate content knowledge from answer-format behavior.
  • BavGround provides a template for localized evaluation below the nation-state level that could be extended to other regional and minority language communities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if Bavarian items were re-translated by native speakers from several Bavarian sub-regions, the measured Bavarian gap would likely widen, because the current single-translator version already biases toward Standard German.
  • Editorial inference: the label-prior diagnostics suggest that some leaderboard positions partly reflect a model's tendency to guess the benchmark's majority answer labels, so future benchmark designers should balance gold-label distributions or report label-prior-adjusted scores.
  • Editorial inference: the semantic-matching protocol depends on an external multilingual embedding model, so protocol comparisons under that view are partly about the embedding model's quality as well as the evaluated LLM's knowledge.
  • Editorial inference: applying the same protocol-aware framework to other region-versus-standard language pairs would test whether protocol sensitivity of this size is a general feature of cultural evaluation or specific to the Bavarian setup.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces BavGround, a multiple-choice benchmark of 206 source questions translated into English, German, and Bavarian (618 instances), spanning eight cultural domains and two item types: general-knowledge (GEN) questions and source-grounded (GRD) questions derived from regional journalism and specialist literature. Fifteen 7B–10B open-weight instruction-tuned models and one closed reference model are evaluated under a protocol-aware framework that includes letter likelihood, shuffled letters, option-text likelihood, generation parsing, semantic matching, and hidden-state diagnostics. The main findings are that, under standard letter scoring, open-weight models average 53.0% accuracy, with Bavarian items 9.8–11.7 points below English and German and GRD items 16.4 points below GEN items; that scores and rankings shift substantially across evaluation protocols; and that a longitudinal analysis of 85 GENBA-10B continued-pretraining checkpoints shows uneven gains across domains, with letter accuracy remaining low while option-text likelihood improves.

Significance. If the empirical claims hold, BavGround is a valuable addition to cultural NLP evaluation, targeting a genuinely underrepresented regional and dialectal setting. Strengths include the multi-parallel design (same source items in three language versions), source-linked GRD items, the use of bootstrap confidence intervals resampled over source IDs for the main letter-scoring gaps, the systematic multi-protocol comparison showing that MCQ evaluation protocols can change model rankings, and the transparent release of artifacts for reproduction. The protocol-sensitivity finding corroborates prior work by Wang et al. (2024) and others and extends it to regional cultural grounding. The checkpoint analysis, though explicitly exploratory, provides a useful demonstration of protocol-divergent learning trajectories. The main weakness is that the headline 'persistent difficulty on Bavarian items' claim is not yet shown to be robust across meaning-based protocols, which is especially consequential given the paper's own methodological message that single-protocol scores can mislead.

major comments (3)
  1. [§5.3, Table 16] The assertion that 'alternative protocols do not erase BAVGROUND's core difficulty structure' is not supported for the Bavarian dimension. Under semantic_embed_generated_answer the open-weight mean is 56.0 (EN) versus 55.7 (BA), a negligible 0.3-point gap, and under option_text_avg the gap narrows to 5.8 points (61.9 versus 56.1); under hidden-state isolated alignment Bavarian is actually the highest language (34.1 versus 33.6 and 33.5). No bootstrap confidence intervals are reported for these alternative-protocol language gaps, whereas the letter-scoring gaps in Appendix Table 8 have them. Because the paper's central methodological lesson is that single-protocol scores can mislead, the headline claim of 'persistent difficulty on Bavarian items' (Abstract and Section 5.1) must be either supported by confidence intervals for meaning-based protocols, or explicitly restricted to letter scoring. The GRD gap appears more robust for GENBA (Table 18), but the Bavarian-language claim as currently stated is letter-scoring-specific.
  2. [§8, Limitations (sixth point)] The Bavarian translations were produced by a single native Chiemgau speaker who has resided outside Bavaria for an extended period, and the paper acknowledges likely dialect attrition, a low Levenshtein distance from German, and the absence of inter-translator agreement. This is a meaningful gap for a benchmark whose central contribution is dialect competence. The caveat that Bavarian performance is an 'upper bound' appears only in the Limitations and is absent from the Abstract and Results. I recommend moving this caveat into the main text and the abstract, and providing at least a few example Bavarian items in the appendix so that reviewers and readers can judge the authenticity of the target variety. Absent such evidence, the dialect-competence claims should be presented as provisional.
  3. [§5.2 and Appendix Table 8] The GEN–GRD gap is reported with a bootstrap confidence interval only under letter scoring (16.4 points, CI [8.2, 24.3]). Given the paper's emphasis on protocol sensitivity, the reader cannot tell whether the 'source-grounded questions are harder' result is stable across protocols for the open-weight mean; the GENBA case study (Table 18) shows the gap persists for that model, but the open-weight mean result is missing. I recommend reporting the open-weight mean GEN–GRD gap under option_text_avg and semantic matching with confidence intervals, or explicitly stating that this cross-protocol analysis is unavailable.
minor comments (5)
  1. [§5.3, Table 16] The phrase 'Bavarian remains below English and German under most strategies' is contradicted by the hidden-state isolated row, where Bavarian (34.1) is highest; please revise to be precise, for example 'under letter and option-text scoring'.
  2. [Table 16] The rows labeled 'Hidden-state contextual' and 'Hidden-state isolated' are described as 'average open-weight accuracy' but are alignment diagnostics; clarify that they are not accuracy measures.
  3. [§1 and §3.2] The translation process is described as performed by 'a native-speaking co-author' in the introduction and by 'an in-house expert and native speaker' in Section 3.2; clarify whether these refer to the same individual.
  4. [Appendix Table 8] The GEN–GRD confidence interval [8.2, 24.3] is wide; consider reporting the bootstrap distribution or a one-sided bound to give readers a better sense of precision.
  5. [Abstract] The phrase 'persistent difficulty with dialectal and localized cultural knowledge' could be read as implying that both components are protocol-robust; consider rewording to reflect the evidence presented in the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BavGround is an empirical benchmark; its main claims are measurements over external models, and the same-author GENBA case study is code-reproduced and not load-bearing.

full rationale

BavGround is an empirical evaluation paper rather than a derivation, so the circularity burden is low and no step reduces a claimed result to its inputs. The benchmark items (Section 3.1) are either LLM-generated with expert validation or hand-built from cited sources; no model output is used to construct or fit the benchmark, so no fitted-input-called-prediction pattern exists. The headline claims regarding Bavarian items and GEN versus GRD gaps are measurements over fifteen open-weight models and one closed reference (Table 2), with bootstrap CIs reported for the main comparisons (Section 4.4 and Appendix Table 8). The same-author GENBA-10B (Hoffmann et al., 2025) is used as one evaluation target and as a checkpoint case study (Sections 4.1 and 5.4); the checkpoint runs are stored as 630,360 per-item records (Appendix C), so the cited model is code-reproduced rather than an unverified authority, and omitting GENBA from the 15-model mean would not change the direction of the reported gaps. The acknowledged limitations are validity threats rather than circularities: Section 8's third limitation (Claude-generated GEN items may overlap with post-training data) could inflate GEN scores, and its sixth limitation (single translator, possible dialect attrition, low Levenshtein distance) is disclosed as making the Bavarian gap an upper bound. Both limitations cut against the headline rather than forcing it. The robustness concern that the Bavarian gap narrows under semantic and hidden-state protocols (Appendix Table 16) and that bootstrap intervals are only reported for letter scoring is a statistical-support issue for the correctness assessment, not a demonstration that a prediction is equivalent to its input. I therefore find no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the benchmark being a valid instrument for cultural competence and the Bavarian translations being authentic. These are domain assumptions rather than mathematical axioms; the paper explicitly discusses threats to both. No free parameters are fitted to produce the results, and no new theoretical entities are introduced. The GENBA checkpoint analysis is self-referential because the model is from the same group, which is why the circularity score is a 2.

assumptions (5)
  • domain assumption Accuracy on multiple-choice questions is a valid operationalization of cultural competence for the claims about regional knowledge.
    Used throughout the evaluation; BavGround treats correct option selection as cultural knowledge. The paper mitigates this by evaluating multiple protocols, but the benchmark score is still a proxy.
  • domain assumption The Bavarian translations are authentic enough to test dialect competence.
    Section 3.2 states translations were produced manually by a native-speaking co-author; Section 8 notes a single translator with extended absence from Bavaria and possible dialect attrition, so this assumption is partially weakened.
  • domain assumption Source-grounded (GRD) questions are unlikely to appear frequently in LLM pretraining corpora, making them a valid probe of deep regional knowledge.
    Section 3.1.2 asserts this, but the paper itself shows GRD items are linked to public sources and are missed by all models, so the assumption is about corpus coverage rather than verified.
  • domain assumption Semantic matching with paraphrase-multilingual-mpnet-base-v2 reliably maps generated answers to options.
    Section 4.3 and Appendix C use this external embedding model to define semantic_embed_generated_answer; results depend on this tool's similarity judgments.
  • domain assumption The GENBA-10B checkpoint trajectories are informative about continued-pretraining effects on cultural knowledge.
    Section 5.4 and Section 8 state the analysis is exploratory and that capacity-matched control runs are needed; the paper treats the trajectories as diagnostic rather than causal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian." pith.science (2026). https://pith.science/paper/ZQR7EKGW

@misc{pith2026260812894,
  author       = {Pith},
  title        = {Pith review of: BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZQR7EKGW}},
  note         = {Machine review of arXiv:2608.12894}
}
read the original abstract

Cultural evaluation of large language models (LLMs) often focuses on high-resource standard languages, leaving regional culture and dialect communities underrepresented. We introduce BavGround, a benchmark for evaluating Bavarian regional cultural grounding and dialect competence across English, German and Bavarian. BavGround contains 206 multiple-choice source questions across eight cultural domains per language, yielding 618 multi-parallel instances, with items covering both broadly accessible cultural knowledge and source-grounded regional knowledge from journalism, historical sources, and specialist literature. We evaluate fifteen 7B-10B open-weight instruction-tuned models and one closed-model reference. Strong multilingual models perform best overall, but performance drops on Bavarian items and source-grounded questions, indicating persistent difficulty with dialectal and localized cultural knowledge. We further show that conclusions depend strongly on evaluation protocol: raw answer-letter scoring, shuffled-letter scoring, option-text likelihood, generated-answer parsing, and semantic matching can produce different absolute scores and rankings, especially for regionally adapted models. Finally, an exploratory analysis of GENBA-10B checkpoints suggests that continued pretraining improves answer-content likelihood unevenly across domains, while dialect competence remains comparatively weak. BavGround supports localized, protocol-aware evaluation of cultural representation in LLMs.

Figures

Figures reproduced from arXiv: 2608.12894 by the authors.

Figure 1
Figure 1. GENBA-10B continued-pretraining accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. GENBA-10B checkpoint trajectories by evaluation strategy, aggregated across all languages and categories. [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗
Figure 3
Figure 3. GENBA-10B checkpoint trajectories by category and evaluation strategy, using all available checkpoints. [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: GENBA-10B checkpoint trajectories for English items by category and evaluation strategy. [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: GENBA-10B checkpoint trajectories for German items by category and evaluation strategy. [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: GENBA-10B checkpoint trajectories for Bavarian items by category and evaluation strategy. [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Interactive dashboard for inspecting BAVGROUND evaluation outputs across models, checkpoints, languages, domains, and scoring strategies [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

81 extracted references · 41 canonical work pages

  1. [1]

    Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298--306

    2021

  2. [3]

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610--623

    2021

  3. [4]

    Jan Betley, Niels Warncke, Anna Sztyber-Betley, Daniel Tan, Xuchan Bao, Mart \' n Soto, Megha Srivastava, Nathan Labenz, and Owain Evans. 2026. Training large language models on narrow tasks can lead to broad misalignment. Nature, 649(8097):584--589

    work page 2026

  4. [5]

    Kirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha, Shachi Dave, and Danish Pruthi. 2026. Tales: A taxonomy and analysis of cultural representations in llm-generated stories. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1--26

    work page 2026

  5. [7]

    Minh Duc Bui, Carolin Holtermann, Valentin Hofmann, Anne Lauscher, and Katharina von der Wense. 2025. Large language models discriminate against speakers of german dialects. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8223--8251

    work page 2025

  6. [9]

    Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-D \"u nner. 2024. Questioning the survey responses of large language models. Advances in Neural Information Processing Systems, 37:45850--45878

    2024

  7. [10]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097--1179

    2024

  8. [11]

    Clifford Geertz. 2025. Thick description: Toward an interpretive theory of culture (1973). In Close Reading and Its Alternatives, pages 107--125. Routledge

    work page 1973

Show all 81 references
  1. [12]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th annual meeting of the association for computational linguistics, p...

    2020

  2. [13]

    Michael A Hedderich, Anyi Wang, Raoyuan Zhao, Florian Eichin, Jonas Fischer, and Barbara Plank. 2025. What’s the difference? supporting users in identifying the effects of prompt and model changes through token patterns. In Proceedings of the 63rd Annual Meeting of the Associa...

    2025

  3. [14]

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam De Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, and 1 others. 2022. Challenges and strategies in cross-cultural nlp. In Proceedings of the 60th Annu...

    2022

  4. [15]

    Michael Herzfeld. 2020. The poetics of manhood: contest and identity in a cretan mountain village

    2020

  5. [17]

    Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural ...

    2021

  6. [20]

    Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2025. Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art. Transactions of the Association for Computational Linguistics, 13:652--689

    2025

  7. [21]

    Danfeng Liu. 2021. Making Heimat in the modern world: state, Catholicism, and nature in a Bavarian village community. Ph.D. thesis, London School of Economics and Political Science

    2021

  8. [23]

    Francesca Merlan. 2004. Preserving the farm in southern germany. Culture & agriculture, 26(1-2):124--136

    2004

  9. [24]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933--949

    2024

  10. [25]

    Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki A Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew A Ayele, and 1 others. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. Advances in Neural Infor...

    2024

  11. [26]

    Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 16366--16393

    2024

  12. [27]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006--2017

    2024

  13. [28]

    Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages...

    2019

  14. [29]

    Donya Rooein, Vil \'e m Zouhar, Debora Nozza, and Dirk Hovy. 2025. Biased tales: Cultural and topic bias in generating children’s stories. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 52--72

    2025

  15. [30]

    Anthony R Rowley. 2011. Bavarian: Successful dialect or failed language. Handbook of language and ethnic identity. The success-failure continuum in language and ethnic identity efforts, 2:299--309

    2011

  16. [32]

    Andrea Seveso, Daniele Potert \` , Edoardo Federici, Mario Mezzanzanica, and Fabio Mercorio. 2025. Italic: An italian culture-aware natural language benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lin...

    2025

  17. [33]

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rog \'e rio Abreu De Paula, and Diyi Yang. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Lingu...

    2024

  18. [34]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. MPNet : Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems, volume 33, pages 16857--16867

    2020

  19. [35]

    o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. “my answer is c

    Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul R \"o ttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. “my answer is c”: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computat...

    2024

  20. [37]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations, volume 2024, pages 19426--19454

    2024

  21. [38]

    Naitian Zhou, David Bamman, and Isaac L Bleaman. 2025. Culture is not trivia: Sociocultural theory for cultural nlp. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25869--25886

    2025

  22. [39]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  23. [40]

    Publications Manual , year = "1983", publisher =

    1983

  24. [41]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    1981

  25. [42]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  26. [43]

    Dan Gusfield , title =. 1997

    1997

  27. [44]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  28. [45]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  29. [46]

    arXiv preprint arXiv:2505.21693 , year=

    MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs , author=. arXiv preprint arXiv:2505.21693 , year=

  30. [47]

    Computational Linguistics , pages=

    Survey of cultural awareness in language models: Text and beyond , author=. Computational Linguistics , pages=. 2025 , publisher=

    2025

  31. [48]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    What’s the difference? supporting users in identifying the effects of prompt and model changes through token patterns , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  32. [49]

    arXiv preprint arXiv:2504.06011 , year=

    Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi , author=. arXiv preprint arXiv:2504.06011 , year=

    arXiv

  33. [50]

    2021 , school=

    Making Heimat in the modern world: state, Catholicism, and nature in a Bavarian village community , author=. 2021 , school=

    2021

  34. [51]

    Journal of Computational Social Science , volume=

    Assessing political bias in large language models , author=. Journal of Computational Social Science , volume=. 2025 , publisher=

    2025

  35. [52]

    Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=

    Persistent anti-muslim bias in large language models , author=. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=

    2021

  36. [53]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Large language models discriminate against speakers of German dialects , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  37. [54]

    Cluster Computing , volume=

    Foundation and large language models: fundamentals, challenges, opportunities, and social impacts , author=. Cluster Computing , volume=. 2024 , publisher=

    2024

  38. [55]

    Nature , volume=

    Training large language models on narrow tasks can lead to broad misalignment , author=. Nature , volume=. 2026 , publisher=

    2026

  39. [56]

    arXiv preprint arXiv:2412.16339 , year=

    Deliberative alignment: Reasoning enables safer language models , author=. arXiv preprint arXiv:2412.16339 , year=

    arXiv

  40. [57]

    Computational Linguistics , volume=

    Bias and fairness in large language models: A survey , author=. Computational Linguistics , volume=. 2024 , publisher=

    2024

  41. [58]

    arXiv preprint arXiv:2503.11985 , year=

    No LLM is Free From Bias: A Comprehensive Study of Bias Evaluation in Large Language models , author=. arXiv preprint arXiv:2503.11985 , year=

    arXiv

  42. [59]

    arXiv preprint arXiv:2509.05668 , year=

    Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian , author=. arXiv preprint arXiv:2509.05668 , year=

    arXiv

  43. [60]

    arXiv preprint arXiv:2005.14050 , year=

    Language (technology) is power: A critical survey of" bias" in nlp , author=. arXiv preprint arXiv:2005.14050 , year=

    2005 arXiv

  44. [61]

    Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    On the dangers of stochastic parrots: Can language models be too big? , author=. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=

    2021

  45. [62]

    Transactions of the Association for Computational Linguistics , volume=

    Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art , author=. Transactions of the Association for Computational Linguistics , volume=. 2025 , publisher=

    2025

  46. [63]

    Advances in neural information processing systems , volume=

    Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models , author=. Advances in neural information processing systems , volume=

  47. [64]

    Close Reading and Its Alternatives , pages=

    Thick Description: Toward an Interpretive Theory of Culture (1973) , author=. Close Reading and Its Alternatives , pages=. 2025 , publisher=

    1973

  48. [65]

    Handbook of language and ethnic identity

    Bavarian: Successful dialect or failed language , author=. Handbook of language and ethnic identity. The success-failure continuum in language and ethnic identity efforts , volume=

  49. [66]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Challenges and strategies in cross-cultural NLP , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  50. [67]

    2020 , publisher=

    The poetics of manhood: contest and identity in a Cretan mountain village , author=. 2020 , publisher=

    2020

  51. [68]

    Culture & agriculture , volume=

    Preserving the farm in Southern Germany , author=. Culture & agriculture , volume=. 2004 , publisher=

    2004

  52. [69]

    ITALIC: An Italian culture-aware natural language benchmark , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  53. [70]

    Artificial Intelligence , volume=

    Quantifying and alleviating political bias in language models , author=. Artificial Intelligence , volume=. 2022 , publisher=

    2022

  54. [71]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    arXiv

  55. [72]

    kelly is a warm person, joseph is a role model

    " kelly is a warm person, joseph is a role model": Gender biases in llm-generated reference letters , author=. arXiv preprint arXiv:2310.09219 , year=

    arXiv

  56. [73]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Having beer after prayer? measuring cultural bias in large language models , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  57. [74]

    My answer is C

    “My answer is C”: First-token probabilities do not match text answers in instruction-tuned language models , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  58. [75]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Culture is not trivia: Sociocultural theory for cultural nlp , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  59. [76]

    Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    Culturebank: An online community-driven knowledge base towards culturally aware language technologies , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=

    2024

  60. [77]

    Advances in Neural Information Processing Systems , volume=

    Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages , author=. Advances in Neural Information Processing Systems , volume=

  61. [78]

    Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , pages=

    Tales: A taxonomy and analysis of cultural representations in llm-generated stories , author=. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems , pages=

    2026

  62. [79]

    arXiv preprint arXiv:2503.01493 , year=

    Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting , author=. arXiv preprint arXiv:2503.01493 , year=

  63. [80]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Biased tales: Cultural and topic bias in generating children’s stories , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  64. [81]

    Transactions of the Association for Computational Linguistics , volume=

    State of what art? a call for multi-prompt llm evaluation , author=. Transactions of the Association for Computational Linguistics , volume=. 2024 , publisher=

    2024

  65. [82]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Don’t stop pretraining: Adapt language models to domains and tasks , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  66. [83]

    Advances in Neural Information Processing Systems , volume=

    Questioning the survey responses of large language models , author=. Advances in Neural Information Processing Systems , volume=

  67. [84]

    and Plank, Barbara and Kreuter, Frauke

    Ma, Bolei and Wang, Xinpeng and Hu, Tiancheng and Haensch, Anna-Carolina and Hedderich, Michael A. and Plank, Barbara and Kreuter, Frauke. The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Models. Findings of the Association for Compu...

    2024 doi

  68. [85]

    arXiv preprint arXiv:2510.11586 , year=

    Survey Response Generation: Generating Closed-Ended Survey Responses In-Silico with Large Language Models , author=. arXiv preprint arXiv:2510.11586 , year=

    arXiv

  69. [86]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , booktitle =. Sentence-. 2019 , publisher =

    2019

  70. [87]

    Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan , booktitle =

  71. [88]

    arXiv preprint arXiv:2507.07188 , year=

    Prompt perturbations reveal human-like biases in llm survey responses , author=. arXiv preprint arXiv:2507.07188 , year=

  72. [89]

    Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    Large language models sensitivity to the order of options in multiple-choice questions , author=. Findings of the Association for Computational Linguistics: NAACL 2024 , pages=

    2024

  73. [90]

    International Conference on Learning Representations , volume=

    Large language models are not robust multiple choice selectors , author=. International Conference on Learning Representations , volume=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.