Pith. sign in

REVIEW 3 major objections 6 minor 56 references

When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Everyday trivia exposes a systematic gap in what LLMs know, in a new multilingual benchmark.

desk verdict A genuinely useful multilingual trivia benchmark with a robust central finding; the cross-lingual claims need translation-provenance work, but the paper deserves a serious referee. read the letter →

arxiv 2607.21445 v1 pith:KX5G57ME submitted 2026-07-23 cs.CL

Anna Mosolova , Djamé Seddah This is my paper
classification cs.CL
keywords multilingualbenchmarksfactualknowledgeeverydaypopularculturelong-taillargelanguagemodelsquiz-styleevaluationcross-lingualconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TriviaRoomQA, a quiz-style benchmark of 3,300 parallel multiple-choice questions in six European languages plus 5,340 French-only questions, covering everyday and long-tail topics from celebrity news to geography. Evaluating 30 open-weight language models, it aims to show that LLM factual knowledge is strong where encyclopedic sources are rich—history, geography, mathematics—but noticeably weaker on popular culture, music, movies, and news. The results also suggest that access to the same factual knowledge is not language-independent: several models answer the same underlying question correctly in one language and incorrectly in another. The authors argue that this everyday-knowledge gap is not captured by existing academic benchmarks, which have become saturated, and that trivia-style questions provide a finer-grained diagnostic for what models actually know. A pilot human comparison adds that humans degrade gradually as questions get harder, while models fall to near-random levels once a topic lies outside their knowledge.

What carries the argument

TRIVIAROOMQA, a quiz benchmark built from human-written trivia questions, each with four candidate answers and annotations for topic, category, difficulty, time period, and geographic region. Its multilingual part contains 3,300 parallel questions in six European languages, so the same underlying fact is asked the same way in English, French, Italian, Spanish, German, and Dutch. That parallel structure lets the authors separate 'what the question asks' from 'what language it is asked in'; answers are scored by picking whichever of the four options the model considers most probable.

What would settle it

Have native speakers of each of the six languages rate the parallel questions for translation fidelity and difficulty, and re-run the evaluation after replacing any non-equivalent items; if accuracy gaps across languages persist on the cleaned set, the language-dependence claim survives, and if they shrink or vanish, it is an artifact of the question set.

Watch

Extended reading notes

Core claim

TriviaRoomQA is derived from human-written quiz questions designed for trivia rooms and shows. Each question has four options, a difficulty level, and metadata for topic, category, time period, and geographic region. Using a scoring method that picks whichever answer option the model considers most probable, the authors find that models' accuracy on historical, geographic, and mathematical questions can be roughly double their accuracy on music, movies, people, and news categories; for 18 of 30 models, those popular-culture categories stay below 50%. They further show that same-question accuracy varies across English, French, Italian, Spanish, German, and Dutch, and that adding a web-search

Load-bearing premise

The cross-lingual conclusion rests on the 3,300 parallel questions being faithful, equally difficult translations across all six languages; the paper does not describe how the translations were produced or checked, so unequal translations could create the appearance of language-dependent knowledge.

Editorial extensions

If this is right

  • Academic benchmarks that focus on expert or school-level knowledge can report high scores even while models fail everyday cultural facts, so TriviaRoomQA offers a complementary diagnostic.
  • Larger models improve absolute accuracy but do not close the popular-culture or cross-lingual gaps, meaning scale alone is not a fix.
  • Because performance varies across languages for identical questions, evaluating models in English alone can misrepresent their factual knowledge for other languages.
  • Giving models access to web search substantially improves performance on this benchmark, indicating that many failures are retrieval failures rather than reasoning failures.
  • Difficulty labels have different meanings for humans and models: on low-knowledge topics, models sit near random across all difficulty levels instead of degrading gradually.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference the paper does not draw: the benchmark could double as a practical testbed for knowledge-editing and retrieval-augmented methods, showing whether the popular-culture gap can be closed rather than merely measured.
  • The cross-lingual inconsistency hints that a fact's internal representation may be tied to the language in which it appears most often in training data; extending the benchmark to less-resourced languages could sharpen this picture.
  • The temporal effect—models do better on older and timeless facts—suggests recency and repetition in training data, not reasoning ability, may be the controlling variable; a controlled frequency analysis with a transparent corpus could test this directly.
  • If the finding generalizes, multilingual product decisions should treat factual reliability as language-dependent: a model may be trustworthy for history in one language but not for celebrity news in another.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. TRIVIAROOMQA is a new quiz-style benchmark sourced from OpenQuizzDB, comprising 3,300 parallel multiple-choice questions in French, English, Italian, Spanish, German, and Dutch, plus 5,340 French-only questions. Questions are annotated with topics, categories, difficulty levels, and approximate continent/time-period labels produced by Llama-70B-Instruct. The authors evaluate 30 open-weight LLMs (7B–70B) using log-likelihood scoring via lm-eval. Their main positive finding is a robust gap between encyclopedic categories (history, geography, math) and everyday popular-culture categories (celebrities, music, movies, news), which persists across model families and scales. They also report cross-lingual accuracy differences for the same parallel questions, a temporal gradient (historical/timeless questions easier than recent ones), and a pilot human study suggesting humans degrade gradually with difficulty while LLMs do not on unfamiliar topics.

Significance. The benchmark fills a real gap: existing English-centric trivia and knowledge benchmarks (SimpleQA, TriviaQA, Natural Questions) lack fine-grained topic annotations and multilingual parallel coverage. The encyclopedic-vs-popular-culture gap is a simple, falsifiable finding that is consistent across 30 models and two benchmark subsets, and it is not an artifact of fitted parameters. The paper also ships reproducible evaluation tasks for lm-eval, releases the data under CC BY-SA, and gives a detailed model inventory. If the cross-lingual and temporal/continental analyses are supported by validation (translation parity, annotation quality, uncertainty quantification), this would be a useful diagnostic resource for knowledge access in LLMs. At present, the core category-level finding is strong, but the multilingual contribution is under-supported.

major comments (3)
  1. [Section 3, Section 5.1] The paper states that TRIVIAROOMQA-MULTI 'consists of 3,300 parallel questions' in six languages, but it never describes how the non-French versions were produced, translated, or checked. Since OpenQuizzDB is a French platform, the other five languages are presumably translations, yet there is no mention of human vs. machine translation, translator expertise, back-translation, or equivalence checks. Section 5.1 interprets cross-language accuracy differences (e.g., ~0.07–0.10 accuracy drops) as evidence that 'factual knowledge is not accessed in a fully language-independent way.' That inference is only valid if the parallel items are semantically and pragmatically equivalent. Translation artifacts (e.g., a calque that reveals the correct option, or an awkward paraphrase that lowers all option likelihoods) could produce exactly the observed pattern. This is the most load-bearing weakness b
  2. [Sections 5.1–5.5] The paper does not report confidence intervals or significance tests for any of its comparative claims. For example, Section 5.1 says Granite-8B and Llama-3.1-8B 'drop by around 0.07 accuracy points' in non-English languages, and Section 5.4 concludes that historical and timeless questions are easier than recent ones. These claims are based on point estimates, and several subsets are small (e.g., Math has 90 questions, Chemistry 60, 2020s period has 195 questions in Table 4). Without uncertainty quantification, it is difficult to know whether the observed cross-lingual, category, and temporal gaps are reliable or within sampling noise. I ask for bootstrap confidence intervals or appropriate significance tests for the main comparisons: language pairs, the encyclopedic vs. popular-culture category contrast, and the time-period gradient.
  3. [Table 2, Sections 5.4–5.5] The continent and time-period annotations were produced automatically by Llama-70B-Instruct, as stated in Table 2 and the Limitations section. The paper correctly labels them as approximate, but then uses them to draw substantive conclusions about performance by continent and time period. No validation of these labels is reported (e.g., a human-annotated sample, agreement rates, or error analysis). Since Llama-70B-Instruct is from the same broad model family as several evaluated models (Llama-3.1-8B/70B), annotation errors could correlate with model behavior in a way that biases the temporal and continental trends. At minimum, the authors should provide a sample-based human evaluation of the metadata labels and report per-label accuracy or agreement.
minor comments (6)
  1. [Section 5.3, Appendix C] The human comparison study has only five participants and is appropriately called a pilot, but the conclusion (Section 6) states as a finding that 'what models find difficult does not always align with what humans find difficult.' Please soften this to reflect the pilot nature, or add more participants.
  2. [Figure 1 and Table 10] The star notation in Figure 1 indicates models trained on a given language, but Table 10 lists language coverage as '-' for some models (e.g., Ministral-8B, Velvet-14B, Qwen models). Please clarify how language training status was determined for these models, or mark them as unknown.
  3. [Section 4.1, Table 10] Model names are inconsistent between the text and the tables: 'sabia-7b' appears in Table 10 but the model list writes 'Sabiá-7B'; 'OLMo-3-7B' in the text is listed as 'Olmo-3-7B-Instruct' in Table 10. Please standardize naming and the use of instruct suffix.
  4. [Section 2, Table 1] The comparison with existing benchmarks is limited to English trivia datasets. Since the paper's contribution is multilingual, position TRIVIAROOMQA against existing multilingual QA benchmarks (e.g., XQuAD, TyDiQA) and clarify the distinction between extractive QA and multiple-choice trivia.
  5. [Table 4, Section 5.4] The time-period table shows that the '2010s' bucket contains 1,601 questions while '2020s' contains only 195. The aggregated '2000-2020s' trend in Figure 4 is therefore dominated by 2000s/2010s items, and the 'recent' category is heterogeneous. The text should note this imbalance when interpreting the temporal gradient.
  6. [Appendix I] The evaluation used one run per model with batch size 16. Since log-likelihood scoring is deterministic, this is fine; but please state that no sampling variance is involved, so that readers do not expect standard-error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TriviaRoomQA is an externally sourced, human-written benchmark, and the paper's claims are measurements over it, not predictions derived from fitted inputs.

full rationale

The paper's central claims are empirical measurements on an externally sourced, human-written quiz dataset (OpenQuizzDB) and do not involve fitted parameters, derived quantities, or an internally constructed target. The category, difficulty, and cross-lingual analyses are direct accuracy measurements over the benchmark's own human-authored labels. The only internally generated artifacts are the continent and time-period metadata, produced by Llama-70B-Instruct and explicitly labeled 'approximate' rather than 'gold-standard' (Section 3, Limitations). These metadata are used only for high-level trend analysis (Sections 5.4, 5.5) and are not load-bearing for the paper's main claims about popular-culture gaps or cross-lingual inconsistency. There is no self-citation chain invoked as proof, no uniqueness theorem imported from the authors, and no fitted input renamed as a prediction. The absence of a described translation pipeline for the 3,300 'parallel' questions is a validity threat to the cross-lingual claim, but it is a data-provenance issue, not a circularity. The paper is self-contained as a benchmark-and-evaluation contribution, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters or invented constructs. The benchmark is a measurement artifact built from external human-written quizzes. The analysis depends on four domain assumptions (translation parity, scoring validity, difficulty-label consistency, LLM-generated metadata quality) and one contamination-bounding assumption; these are the main sources of uncertainty in the paper's fine-grained claims.

assumptions (5)
  • domain assumption The 3,300 questions are semantically parallel across the six languages, with equivalent difficulty and register.
    Section 3 introduces the parallel set without describing translation or validation; all cross-lingual comparisons assume equivalence.
  • domain assumption Conditional log-likelihood scoring of the four answer options is a valid measure of factual knowledge with no option-length or phrasing bias.
    Section 4 specifies lm-eval log-likelihood selection; this is standard practice but known to be sensitive to prompt formatting and option verbalization.
  • domain assumption The OpenQuizzDB difficulty labels (beginner/confirmed/expert) are meaningful and consistent enough to serve as an independent variable for model-vs-human difficulty curves.
    Section 5.3 interprets accuracy by difficulty level; the paper does not validate the labels against human-judged difficulty beyond the n=5 pilot.
  • ad hoc to paper Llama-70B-Instruct-generated continent and time-period labels are accurate enough for trend-level analysis.
    Section 3: 'annotations * were produced automatically using Llama-70B-Instruct'; the paper acknowledges they may contain errors and reports no human agreement.
  • domain assumption OpenQuizzDB content is not substantially present in the models' training corpora.
    Section 3 checks FineWeb2 (214 mentions) and Infini-Gram topic frequency for OLMo-3, which bounds contamination only for those two corpora; coverage in other training data is unknown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs." pith.science (2026). https://pith.science/paper/KX5G57ME

@misc{pith2026260721445,
  author       = {Pith},
  title        = {Pith review of: When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KX5G57ME}},
  note         = {Machine review of arXiv:2607.21445}
}
read the original abstract

Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics. We introduce TriviaRoomQA, a multilingual benchmark designed to evaluate everyday, culturally grounded, and long-tail knowledge across 288 topics. The benchmark contains 3,300 parallel multiple-choice questions in six European languages and additional 5,340 French-only questions for a more fine-grained case study. We evaluate 30 open-weight LLMs from European, Asian, and North American providers, covering models from 7 to 70B parameters. We find that models are strong on knowledge-intensive topics such as history, geography, and mathematics, but substantially weaker on everyday popular-culture topics such as celebrities, music, movies, and news. Moreover, model performance varies across languages even for the same underlying questions, suggesting that access to factual knowledge is not always language-independent. In sum, our dataset and experiments demonstrate an important knowledge gap which is not captured by existing academic-based saturated benchmarks.

Figures

Figures reproduced from arXiv: 2607.21445 by the authors.

Figure 1
Figure 1. Model performance on the six languages of [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Model performance on 24 categories from the TRIVIAROOMQA-FRENCH. The reported metric is accuracy; greener cells indicate higher scores. Models are sorted horizontally from left to right by performance within each region, and categories are sorted vertically from top to bottom by average performance across all models. ⋆ next to a score indicates that the model was trained on French. ▲ marks models trained on 10T+ tok… view at source ↗
Figure 3
Figure 3. Average human and model performance on the six easiest and six hardest topics for models in TRIVIAROOMQA-FRENCH, grouped by question diffi￾culty. To further examine whether models and humans react similarly to difficulty, we selected the six eas￾iest and six hardest topics for models, according to their average performance, and asked five French speakers to answer the corresponding questions.11 [PITH_FULL_IMAGE:fig… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Model performance on TRIVIAROOMQA￾FRENCH grouped by the time period to which each question is most closely related, when applicable. The reported metric is accuracy, greener cells indicate higher scores. ▲ marks models trained on 10T+ tokens. Model size is color-coded:…
Figure 6
Figure 6. Figure 6: Model performance on TRIVIAROOMQA￾FRENCH, grouped by question difficulty. The reported metric is accuracy; greener cells indicate higher scores [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Model performance with chat templates on TRIVIAROOMQA-FRENCH, grouped by question diffi￾culty. The reported metric is accuracy [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 9
Figure 9. Figure 9: Model performance on TRIVIAROOMQA-FRENCH, grouped by the time period to which each question is most closely related, when applicable. The reported metric is accuracy [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Model performance on TRIVIAROOMQA-FRENCH, grouped by the continent to which each question is most closely related, when applicable. Models are grouped by continent of origin, with the average performance for each group reported in the corresponding AVG-{continent} row…
Figure 11
Figure 11. Figure 11: Model performance with chat templates on 24 categories from TRIVIAROOMQA-FRENCH. The reported metric is accuracy. Models are sorted horizontally from left to right by performance within each region, and categories are sorted vertically from top to bottom by average pe…
Figure 12
Figure 12. Figure 12: Model performance with chat templates on TRIVIAROOMQA-FRENCH, grouped by the time period to which each question is most closely related, when applicable. The reported metric is accuracy [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Model performance with chat templates on TRIVIAROOMQA-FRENCH, grouped by the continent to which each question is most closely related, when applicable. Models are grouped by continent of origin, with the average performance for each group reported in the corresponding…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 2 canonical work pages

  1. [1]

    Head-to-Tail: How Knowledgeable are Large Language Models ( LLM s)? A

    Sun, Kai and Xu, Yifan and Zha, Hanwen and Liu, Yue and Dong, Xin Luna. Head-to-Tail: How Knowledgeable are Large Language Models ( LLM s)? A . K . A . Will LLM s Replace Knowledge Graphs?. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)....

    doi:10.18653/v1/2024.naacl-long.18 2024

  2. [2]

    C ultural B ench: A Robust, Diverse and Challenging Benchmark for Measuring LM s' Cultural Knowledge Through Human- AI Red-Teaming

    Chiu, Yu Ying and Jiang, Liwei and Lin, Bill Yuchen and Park, Chan Young and Li, Shuyue Stella and Ravi, Sahithya and Bhatia, Mehar and Antoniak, Maria and Tsvetkov, Yulia and Shwartz, Vered and Choi, Yejin. C ultural B ench: A Robust, Diverse and Challenging Benchmark for Measuring LM s' Cultural Knowledge Through Human- AI Red-Teaming. Proceedings of th...

    doi:10.18653/v1/2025.acl-long.1247 2025

  3. [3]

    and Alsehibani, Arwa and Qandos, Nour and Elshehy, Omar and Abdelkader, Mohamed and Koubaa, Anis

    Nacar, Omer and Sibaee, Serry Taiseer and Ahmed, Samar and Ben Atitallah, Safa and Ammar, Adel and Alhabashi, Yasser and Al-Batati, Abdulrahman S. and Alsehibani, Arwa and Qandos, Nour and Elshehy, Omar and Abdelkader, Mohamed and Koubaa, Anis. Towards Inclusive A rabic LLM s: A Culturally Aligned Benchmark in A rabic Large Language Model Evaluation. Proc...

    2025

  4. [4]

    Having Beer after Prayer? Measuring Cultural Bias in Large Language Models

    Naous, Tarek and Ryan, Michael J and Ritter, Alan and Xu, Wei. Having Beer after Prayer? Measuring Cultural Bias in Large Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.862

    doi:10.18653/v1/2024.acl-long.862 2024

  5. [5]

    BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages , url =

    Antypas, Dimosthenis and Ayele, Abinew and Borkakoty, Hsuvas and Camacho-Collados, Jose and Gutiérrez-Basulto, Víctor and Ibáñez-García, Yazmín and Jin, Jiho and Kim, Eunsu and Lee, Hwaran and Lee, Nayeon and Muhammad, Shamsuddeen and Myung, Junho and Oh, Alice and Ousidhoum, Nedjma and Park, Kiwoong and Perez-Almendros, Carla and Pilehvar, Mohammad and P...

  6. [6]

    2026 , eprint=

    CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks , author=. 2026 , eprint=

    2026

  7. [7]

    How People Use ChatGPT , url =

    Chatterji, Aaron and Cunningham, Thomas and Deming, David and Hitzig, Zoe and Ong, Christopher and Shan, Carl Yan and Wadman, Kevin , year =. How People Use ChatGPT , url =. doi:10.3386/w34255 , institution =

    doi:10.3386/w34255

  8. [8]

    Introducing

    OpenAI , year = 2022, note =. Introducing

    2022

Show all 56 references
  1. [9]

    2026 , eprint=

    Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications , author=. 2026 , eprint=

    2026

  2. [10]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Pfister, Jan and Wunderle, Julia and Hotho, Andreas. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.111

    2025 doi

  3. [11]

    Proceedings of the International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. Proceedings of the International Conference on Learning Representations (ICLR) , year=

  4. [12]

    2021 , eprint=

    Training Verifiers to Solve Math Word Problems , author=. 2021 , eprint=

    2021

  5. [13]

    and Uszkoreit, Jakob and Le, Quoc and Petrov, Slav

    Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-W...

    2019 doi

  6. [14]

    T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

    Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke. T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ...

    2017 doi

  7. [15]

    2024 , eprint=

    Measuring short-form factuality in large language models , author=. 2024 , eprint=

    2024

  8. [16]

    C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge

    Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan. C ommonsense QA : A Question Answering Challenge Targeting Commonsense Knowledge. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Hu...

    2019 doi

  9. [17]

    2024 , eprint=

    Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking , author=. 2024 , eprint=

    2024

  10. [18]

    BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages , url =

    Myung, Junho and Lee, Nayeon and Zhou, Yi and Jin, Jiho and Putri, Rifki Afina and Antypas, Dimosthenis and Borkakoty, Hsuvas and Kim, Eunsu and Perez-Almendros, Carla and Ayele, Abinew Ali and Guti\'. BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and L...

  11. [19]

    Leveraging W ikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to L atin A merica

    Karmim, Yannis and Pino, Renato and Contreras, Hernan and Lira, Hernan and Cifuentes, Sebastian and Escoffier, Simon and Mart \'i , Luis and Seddah, Djam \'e and Barriere, Valentin. Leveraging W ikidata for Geographically Informed Sociocultural Bias Dataset Creation: Applicati...

    2026 doi

  12. [20]

    The Anthropology of Food: How NLP can Help us Unravel the Food cultures of the World

    Riabi, Arij and Saha, Sougata and Choudhury, Monojit. The Anthropology of Food: How NLP can Help us Unravel the Food cultures of the World. Proceedings of the First Workshop on Multilingual Multicultural Evaluation. 2026. doi:10.18653/v1/2026.mme-main.6

    2026 doi

  13. [21]

    2026 , eprint=

    LPFQA: A Long-Tail Professional Forum-based Benchmark for LLM Evaluation , author=. 2026 , eprint=

    2026

  14. [22]

    2026 , note =

    UNESCO , title =. 2026 , note =

    2026

  15. [23]

    2026 , note =

    Ethnologue , title =. 2026 , note =

    2026

  16. [24]

    2025 , eprint=

    Gaperon: A Peppered English-French Generative Language Model Suite , author=. 2025 , eprint=

    2025

  17. [25]

    Minerva LLM s: The First Family of Large Language Models Trained from Scratch on I talian Data

    Orlando, Riccardo and Moroni, Luca and Huguet Cabot, Pere-Llu \'i s and Conia, Simone and Barba, Edoardo and Orlandini, Sergio and Fiameni, Giuseppe and Navigli, Roberto. Minerva LLM s: The First Family of Large Language Models Trained from Scratch on I talian Data. Proceeding...

    2024

  18. [26]

    Advances in Neural Information Processing Systems , volume=

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark , author=. Advances in Neural Information Processing Systems , volume=

  19. [27]

    2023 , eprint=

    Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models , author=. 2023 , eprint=

    2023

  20. [28]

    H ella S wag: Can a Machine Really Finish Your Sentence?

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin. H ella S wag: Can a Machine Really Finish Your Sentence?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. doi:10.18653/v1/P19-1472

    2019 doi

  21. [29]

    Proceedings of the AAAI Conference on Artificial Intelligence , author=

    PIQA: Reasoning about Physical Commonsense in Natural Language , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2020 , month=. doi:10.1609/aaai.v34i05.6239 , abstractNote=

    2020 doi

  22. [30]

    AGIE val: A Human-Centric Benchmark for Evaluating Foundation Models

    Zhong, Wanjun and Cui, Ruixiang and Guo, Yiduo and Liang, Yaobo and Lu, Shuai and Wang, Yanlin and Saied, Amin and Chen, Weizhu and Duan, Nan. AGIE val: A Human-Centric Benchmark for Evaluating Foundation Models. Findings of the Association for Computational Linguistics: NAACL...

    2024 doi

  23. [31]

    MERA : A Comprehensive LLM Evaluation in R ussian

    Fenogenova, Alena and Chervyakov, Artem and Martynov, Nikita and Kozlova, Anastasia and Tikhonova, Maria and Akhmetgareeva, Albina and Emelyanov, Anton and Shevelev, Denis and Lebedev, Pavel and Sinev, Leonid and Isaeva, Ulyana and Kolomeytseva, Katerina and Moskovskiy, Daniil...

    2024

  24. [32]

    and Levin, John-Clark and Kazakov, Mstyslav and Feng, Fiona and Feng, Steven Y

    Phan, Long and Gatti, Alice and Li, Nathaniel and Khoja, Adam and Kim, Ryan and Ren, Richard and Hausenloy, Jason and Zhang, Oliver and Mazeika, Mantas and Hendrycks, Dan and Han, Ziwen and Hu, Josephina and Zhang, Hugh and Zhang, Chen Bo Calvin and Shaaban, Mohamed and Ling, ...

    2026

  25. [33]

    When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

    Mallen, Alex and Asai, Akari and Zhong, Victor and Das, Rajarshi and Khashabi, Daniel and Hajishirzi, Hannaneh. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. Proceedings of the 61st Annual Meeting of the Association f...

    2023 doi

  26. [34]

    2025 , eprint=

    FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language , author=. 2025 , eprint=

    2025

  27. [35]

    2025 , eprint=

    EuroLLM-9B: Technical Report , author=. 2025 , eprint=

    2025

  28. [36]

    2026 , eprint=

    EuroLLM-22B: Technical Report , author=. 2026 , eprint=

    2026

  29. [37]

    arXiv preprint arXiv:2401.17377 , year=

    Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens , author=. arXiv preprint arXiv:2401.17377 , year=

    arXiv

  30. [38]

    doi:10.5281/zenodo.12608602 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and...

    doi

  31. [39]

    Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rémi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu an...

    2020

  32. [40]

    2025 , eprint=

    Apertus: Democratizing Open and Compliant LLMs for Global Language Environments , author=. 2025 , eprint=

    2025

  33. [41]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  34. [42]

    2025 , eprint=

    Salamandra Technical Report , author=. 2025 , eprint=

    2025

  35. [43]

    2026 , eprint=

    TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representation , author=. 2026 , eprint=

    2026

  36. [44]

    2026 , eprint=

    Olmo 3 , author=. 2026 , eprint=

    2026

  37. [45]

    Gemma 3 , url=

    Gemma Team , year=. Gemma 3 , url=

  38. [46]

    Sabiá: Portuguese Large Language Models , ISBN=

    Pires, Ramon and Abonizio, Hugo and Almeida, Thales Sales and Nogueira, Rodrigo , year=. Sabiá: Portuguese Large Language Models , ISBN=. doi:10.1007/978-3-031-45392-2_15 , booktitle=

    doi

  39. [47]

    2024 , eprint=

    Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier , author=. 2024 , eprint=

    2024

  40. [48]

    Alzahrani and Nouf M

    M Saiful Bari and Yazeed Alnumay and Norah A. Alzahrani and Nouf M. Alotaibi and Hisham Abdullah Alyahya and Sultan AlRashed and Faisal Abdulrahman Mirza and Shaykhah Z. Alsubaie and Hassan A. Alahmed and Ghadah Alabduljabbar and Raghad Alkhathran and Yousef Almushayqih and Ra...

    2025

  41. [49]

    2025 , eprint=

    The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation , author=. 2025 , eprint=

    2025

  42. [50]

    Culture is Not Trivia: Sociocultural Theory for Cultural NLP

    Zhou, Naitian and Bamman, David and Bleaman, Isaac L. Culture is Not Trivia: Sociocultural Theory for Cultural NLP. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.1256

    2025 doi

  43. [51]

    Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLM s

    Kabir, Mohsinul and Abrar, Ajwad and Ananiadou, Sophia. Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLM s. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.2

    2025 doi

  44. [52]

    Culture is Everywhere: A Call for Intentionally Cultural Evaluation

    Oh, Juhyun and Cha, Inha and Saxon, Michael and Lim, Hyunseung and Bhatt, Shaily and Oh, Alice. Culture is Everywhere: A Call for Intentionally Cultural Evaluation. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025. doi:10.18653/v1/2025.findings-emnlp.1043

    2025 doi

  45. [53]

    Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens

    Alkhamissi, Mai and Xiao, Yunze and AlKhamissi, Badr and Diab, Mona T. Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. doi:10.18653/v1/2026.findings-eacl.63

    2026 doi

  46. [54]

    Towards Measuring and Modeling ``Culture'' in LLM s: A Survey

    Adilazuarda, Muhammad Farid and Mukherjee, Sagnik and Lavania, Pradhyumna and Singh, Siddhant Shivdutt and Aji, Alham Fikri and O ' Neill, Jacki and Modi, Ashutosh and Choudhury, Monojit. Towards Measuring and Modeling ``Culture'' in LLM s: A Survey. Proceedings of the 2024 Co...

    2024 doi

  47. [55]

    Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLM s

    Ifergan, Maxim and Choshen, Leshem and Aharoni, Roee and Szpektor, Idan and Abend, Omri. Beneath the Surface of Consistency: Exploring Cross-lingual Knowledge Representation Sharing in LLM s. Findings of the Association for Computational Linguistics: NAACL 2025. 2025

    2025

  48. [56]

    AI Magazine , author=

    Building Watson: An Overview of the DeepQA Project , volume=. AI Magazine , author=. 2010 , month=. doi:10.1609/aimag.v31i3.2303 , abstractNote=

    2010 doi

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.