Pith. sign in

REVIEW 3 major objections 5 minor 44 references

A taxonomy-plus-RAG pipeline synthesizes culturally valid QA data and places a 3B-parameter floor on LLM cultural competence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:26 UTC pith:PDPDPBYA

load-bearing objection A solid framework for cultural QA synthesis, but the 4,149-entry verification claim conflicts with the paper's own 840-sample annotation protocol. the 3 major comments →

arxiv 2509.10886 v1 pith:PDPDPBYA submitted 2025-09-13 cs.CL cs.AI

CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis

Xinyu Zhang , Pei Zhang , Shuang Luo , Jialong Tang , Yu Wan , Baosong Yang , Fei Huang This is my paper
classification cs.CL cs.AI
keywords cultural competenceLLM evaluationsynthetic QA benchmarkretrieval-augmented generationhierarchical taxonomymultilingual3B parameter thresholdmixture of experts
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that cultural competence in large language models can be measured and improved systematically if it is anchored to a comprehensive taxonomy and grounded in retrieved factual knowledge. The authors build CultureSynth, a two-part framework: a hierarchical multilingual taxonomy of 12 primary and 130 secondary cultural topics, and a retrieval-augmented generation pipeline that turns vetted facts into culturally specific question-answer pairs. Using the resulting CultureSynth-7 benchmark—19,360 QA pairs across seven languages—they evaluate 14 LLMs and report a clear ranking, with ChatGPT-4o-Latest on top and open models like Qwen2.5-72B-Instruct close behind. They claim a 3B-parameter threshold is necessary for basic cultural competence, with smaller models collapsing to native-language defaults. They also find that Mixture-of-Experts architectures retrieve cultural knowledge better, dense transformers handle long-context cultural topics better, and every model shows geographic and domain-specific blind spots.

Core claim

The central claim is that a taxonomy-guided, retrieval-augmented synthesis pipeline can produce high-quality, culturally relevant QA data at scale with minimal manual annotation, and that evaluating on this data exposes a reproducible performance hierarchy among LLMs. On the paper's own terms, the key finding is the 3B-parameter threshold: models with fewer than 3 billion parameters cannot sustain coherent answers to culturally specific questions and revert to native-language functionality, while larger models—especially ChatGPT-4o-Latest and Qwen2.5-72B-Instruct—show strong, stratified cultural competence. The paper further claims that architecture matters independently of size: Mixture-of-

What carries the argument

The load-bearing machinery is a two-stage pipeline. First, a hierarchical taxonomy is assembled by merging five library classification systems into 12 primary and 130 secondary cultural topics, then enlarged by an LLM role-playing prompt to yield 300-plus tertiary topics and over 1,000 language-specific keywords per language. Second, a retrieval-augmented generation loop takes each keyword, retrieves and filters encyclopedia pages for cultural relevance, extracts structured knowledge points, and prompts an expert-role LLM to generate self-contained questions and detailed answers in the target language. The taxonomy provides coverage and prevents topic fragmentation; the RAG step anchors answ

Load-bearing premise

The entire quality argument for the benchmark rests on native-speaker verification of only 120 randomly sampled QA pairs per language—840 out of 19,360—so if that sample is unrepresentative, the reported quality scores and the derived model rankings do not generalize to the full dataset.

What would settle it

A concrete test: take a fresh random sample of 500 QA pairs per language from the released CultureSynth-7 (not the same 120), have native speakers score them with the same rubric, and compare acceptance rates to the reported 95.8% clarity, 83.5% relevance, and 98.8% answer quality. If relevance or answer quality drops substantially, the benchmark's validation claim is falsified and the model rankings built on it are called into question.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Below 3B parameters, models cannot answer culturally specific questions reliably, so small-device deployments should not be expected to handle cultural tasks without further training.
  • The proposed pipeline can generate additional cultural QA data for other languages and topics without manual question writing, lowering the cost of expanding cultural benchmarks.
  • Architecture choice matters: Mixture-of-Experts models are better at retrieving discrete cultural facts, while dense models with long contexts are better at law and political-science style questions, so model selection should be task-aware.
  • The reported language- and topic-wise performance gaps (e.g., GPT-4o in East Asian contexts, Claude-3.5 in Arabic and Korean) identify concrete targets for fine-tuning and data collection.
  • The high agreement between two different judge LLMs (around 85%) suggests LLM-as-judge evaluation of cultural competence is stable, enabling cheaper and faster comparisons than human evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 3B threshold is probably not a universal law but an artifact of current training data and architecture; a small model trained specifically on culture-rich multilingual data could plausibly exceed it, which would be a clean, testable extension.
  • Because the taxonomy is built from library classifications, it inherits their Western-centric organizing biases; categories like 'Religion and Theology' are not neutral across cultures, even though the pipeline localizes keywords per language.
  • The same RAG plus expert-role pipeline could be adapted to generate not just QA pairs but culturally grounded dialogue or preference data, which would speak directly to behavioral cultural alignment rather than knowledge recall.
  • If the benchmark is used for fine-tuning, users should prefer the verified subset because the larger unverified set may contain hallucinated or unverified content, and the paper's quality claims rest on a small sample.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces CultureSynth, a framework that combines a hierarchical multilingual cultural taxonomy (12 primary topics, 130 secondary topics) with a RAG-based pipeline for synthesizing culturally relevant QA pairs in seven languages. The authors release CultureSynth-7, described as containing 19,360 synthetic QA pairs and 4,149 manually verified entries, and evaluate 14 LLMs of varying sizes using pairwise LLM-as-a-judge comparisons. They report a clear performance hierarchy, claim a 3B-parameter threshold for basic cultural competence, and analyze language-, topic-, and architecture-specific patterns. The paper also argues that the framework reduces reliance on manual annotation while preserving data quality.

Significance. If the claims hold, the benchmark and synthesis framework would be a useful contribution to multilingual cultural evaluation: the taxonomy built from five library classifications is systematic, the RAG anchoring is a sensible way to reduce hallucination, the benchmark is publicly released, and the two-judge consistency check (85% average agreement) is a positive methodological step. The cross-lingual and cross-topic breakdowns are informative. However, the paper's central claims currently rest on an internally inconsistent verification count and on an overgeneralized parameter-threshold statement, and the evaluation loop has a same-family generator/judge/top-model circularity that needs to be addressed before the conclusions can be taken at face value.

major comments (3)
  1. [§4.1, §4.2, Abstract] The paper claims in the abstract that CultureSynth-7 contains '4,149 manually verified entries', and §4.2 states that 'all questions annotated for clarity and cultural relevance, and answers verified for high quality.' However, §4.1 describes randomly sampling 120 QA pairs per language for annotation, i.e., 120 × 7 = 840 pairs. These two statements cannot both be true. Unless the sampling scheme is a formal certification protocol that statistically guarantees the remaining 3,309 entries, the released 4,149-entry benchmark is not 'manually verified' as claimed. This matters because Tables 3–9 and the 3B-threshold result are computed on the full 4,149-entry set, so the validity of those results depends on the unverified portion. The authors should either (a) provide evidence that the 840-pair sample certifies the full set under an explicit sampling design, or (b) revise all 'manually verif
  2. [§5.3, Abstract, Figure 8] The abstract's claim that 'a 3B-parameter threshold is necessary for achieving basic cultural competence' is an overgeneralization of the evidence. The models below 3B parameters in Figure 8 come from only two families (Qwen2.5 and Llama 3.2), and the evaluation is on a single synthetic benchmark generated by the same LLM family as the top-performing judge. A threshold inferred from this narrow sample cannot be stated as a general necessity for all architectures, training distributions, and cultural tasks. At minimum, the claim should be restricted to the tested models and benchmark, and ideally supplemented with additional small models or a different benchmark before being presented as a general finding.
  3. [§5.1, §5.2, Tables 3 and 8] The evaluation loop is partially self-referential: the benchmark data are generated by a GPT-4-based pipeline, the primary judge is ChatGPT-4o-Latest, and the top-ranked model is ChatGPT-4o-Latest. This does not by itself invalidate the ranking, but it creates a risk that the judge systematically prefers responses that resemble the style and content of the data-generation model, inflating the score of the same-family model. The Qwen3-32B-Think judge provides a useful cross-check, but both judges are LLMs, and the agreement rate (85%) does not establish that either judge is unbiased with respect to the source model. I recommend reporting a human-judged sample of pairwise comparisons and/or a position-swap and reference-perturbation analysis to show that the ranking is robust beyond judge-model self-preference.
minor comments (5)
  1. [§4.1, Table 1] The text reports a question clarity average of 95.41%, while Table 1's per-language averages give approximately 95.8%. The authors should harmonize the numbers.
  2. [Figure 5 and Figure 6 captions] The captions contain a broken glyph: 'Mistral-/glyph1197emo-Instruct' should presumably be 'Mistral-Nemo-Instruct'.
  3. [§4.2, footnote 3] The footnote says 'We use CultureSynth-7 to identify CultureSynth-7-mini, which are manually verified and released publicly.' This directly conflicts with §4.1 and the abstract, and should be clarified or corrected as part of the major revision.
  4. [§3.2, Step 1] The retrieval step mentions excluding pages that fail 'keyword relevance or cultural content requirements,' but the exact thresholds or criteria are not specified. Since RAG grounding is a key claim of the paper, a few sentences on how relevance is operationalized would improve reproducibility.
  5. [Limitations] The limitations section discusses topic imbalance and lack of cognitive-demand taxonomies, but does not mention the verification-count issue or the potential judge-model bias. Both should be acknowledged, or, preferably, addressed with additional analysis.

Circularity Check

1 steps flagged

Partial self-reference in evaluation: GPT-generated benchmark, ChatGPT-4o-Latest as judge and top-ranked model; framework construction itself is not circular.

specific steps
  1. other [Section 3.1; Section 5.2; Section 5.3 (Table 3)]
    "we use GPT-4 (Achiam et al., 2023) ... We select ChatGPT-4o-Latest (version gpt-4o-2024-05-13) (Achiam et al., 2023) and Qwen3-32B-Think (Yang et al., 2025) as our judge LLMs ... the overall ranking of cultural competence follows this order: ChatGPT-4o-Latest > Qwen2.5-72B-Instruct"

    The benchmark's questions and reference answers are produced by GPT-4-family models from Wikipedia, and the primary judge is ChatGPT-4o-Latest, the same model that is then reported as the best performer (Table 3, 76.31% net win rate). The score determining this ranking is assigned by the model being ranked, so the top position is partly self-awarded by construction: judge and contestant coincide, and the test data shares the judge's generation family. No procedure removes judge self-preference; the 85.05% agreement with Qwen3-32B-Think and Wikipedia grounding give partial independence, making the circularity partial rather than total.

full rationale

The core framework is not circular: the taxonomy integrates five external library classification systems (Section 3.1), QA synthesis is grounded in retrieved Wikipedia knowledge (Section 3.2), and the quality check uses native-speaker annotation of a sample (Section 4.1). These inputs are external to the paper's conclusions. The main circularity-adjacent issue is the evaluation loop: GPT-4 generates the data, ChatGPT-4o-Latest judges the models, and ChatGPT-4o-Latest ranks first (Sections 3.1, 5.2, 5.3). This makes the headline ranking partially self-referential, although the second judge (Qwen3-32B-Think) and 85.05% agreement provide some independent content. Separately, a non-circular but load-bearing internal inconsistency must be noted: Section 4.1 annotates only 120 QA pairs per language (840 total), yet the abstract and Section 4.2 claim '4,149 manually verified entries' and state that 'all questions annotated for clarity and cultural relevance, and answers verified for high quality.' The full benchmark's verification status is therefore asserted beyond the described protocol, which weakens the validity of every Section 5 finding built on the 4,149-entry set; this is a missing-support/correctness concern, not a reduction of a prediction to an input. Overall, no definitional or self-citation circularity is present, so the score reflects the partial self-reference in the judge/contestant design rather than a fully forced result.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claims rest on several domain assumptions about source reliability, LLM generation validity, and sample representativeness, with no free fitted parameters or invented physical entities.

axioms (5)
  • domain assumption The five library classification systems (DDC, UDC, LCC, CLC, NDC) together provide comprehensive, universal coverage of cultural topics.
    Section 3.1 builds the taxonomy from these systems without demonstrating that they exhaustively cover non-Western cultural domains.
  • ad hoc to paper LLM-generated tertiary topics and keywords are culturally accurate and representative for each language/country.
    Section 3.1 uses a role-playing prompt with GPT-4 to generate topics and keywords, with no human validation of this content.
  • domain assumption Wikipedia articles in English and target languages are authentic, reliable, and sufficient sources for cultural knowledge.
    Section 3.2 and 4.1: QA generation and annotation reference Wikipedia as ground truth, without external verification.
  • domain assumption LLM-as-judge pairwise comparison against a reference answer gives a valid measure of cultural competence.
    Section 5.1 relies on ChatGPT-4o-Latest and Qwen3-32B-Think as judges; high agreement is shown, but no human validation of judge outputs is reported.
  • ad hoc to paper A sample of 120 QA pairs per language is representative for validating the quality of the full synthetic set.
    Section 4.1: only 840 of 19,360 entries are annotated, yet the abstract claims 4,149 manually verified entries, making this assumption questionable.

pith-pipeline@v1.3.0-alltime-deepseek · 20391 in / 10308 out tokens · 113814 ms · 2026-08-04T17:26:17.650303+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis." pith.science (2026). https://pith.science/paper/PDPDPBYA

@misc{pith2026250910886,
  author       = {Pith},
  title        = {Pith review of: CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDPDPBYA}},
  note         = {Machine review of arXiv:2509.10886}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Cultural competence, defined as the ability to understand and adapt to multicultural contexts, is increasingly vital for large language models (LLMs) in global environments. While several cultural benchmarks exist to assess LLMs' cultural competence, current evaluations suffer from fragmented taxonomies, domain specificity, and heavy reliance on manual data annotation. To address these limitations, we introduce CultureSynth, a novel framework comprising (1) a comprehensive hierarchical multilingual cultural taxonomy covering 12 primary and 130 secondary topics, and (2) a Retrieval-Augmented Generation (RAG)-based methodology leveraging factual knowledge to synthesize culturally relevant question-answer pairs. The CultureSynth-7 synthetic benchmark contains 19,360 entries and 4,149 manually verified entries across 7 languages. Evaluation of 14 prevalent LLMs of different sizes reveals clear performance stratification led by ChatGPT-4o-Latest and Qwen2.5-72B-Instruct. The results demonstrate that a 3B-parameter threshold is necessary for achieving basic cultural competence, models display varying architectural biases in knowledge processing, and significant geographic disparities exist across models. We believe that CultureSynth offers a scalable framework for developing culturally aware AI systems while reducing reliance on manual annotation\footnote{Benchmark is available at https://github.com/Eyr3/CultureSynth.}.

Figures

Figures reproduced from arXiv: 2509.10886 by Baosong Yang, Fei Huang, Jialong Tang, Pei Zhang, Shuang Luo, Xinyu Zhang, Yu Wan.

Figure 1
Figure 1. Figure 1: CultureSynth consists of a hierarchically structured multilingual cultural taxonomy (left of double line) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of CultureSynth: (1) a hierarchically multilingual cultural taxonomy (see Section [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Topic distribution for total 19,360 QA pairs. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Net win rates (%) of different models compared to the baseline model across languages. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Net win rates (%) of different models compared to the baseline model across cultural topics. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Net win rates (%) of models of different sizes [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Net win rates (%) of models under 3B parameters compared to the baseline model across languages. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Net win rates (%) of different models compared to the baseline model across cultural topics. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt for extending primary and secondary topics into tertiary topics and keywords. [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt for determining whether the retrieved page is related to the keyword (Step 1). [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for determining whether the retrieved page contains culture-specific content (Step 1). [PITH_FULL_IMAGE:figures/full_fig_p014_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt for knowledge extraction in different cultural knowledge settings (Step 2). [PITH_FULL_IMAGE:figures/full_fig_p015_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt for knowledge extraction in unique cultural knowledge settings (Step 2). [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt for question generation in different cultural knowledge settings (Step 3). [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Prompt for question generation in unique cultural knowledge settings (Step 3). [PITH_FULL_IMAGE:figures/full_fig_p016_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt for answer generation in both different and unique cultural knowledge settings (Step 3). [PITH_FULL_IMAGE:figures/full_fig_p017_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for reference-guided pairwise comparison ( [PITH_FULL_IMAGE:figures/full_fig_p017_18.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 23 linked inside Pith

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  2. [2]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

    2024

  3. [3]

    Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit Iyyer, and Eunsol Choi. 2024. Calmqa: Exploring culturally specific long-form question answering across 23 languages. CoRR

    2024

  4. [4]

    MJ Bennett. 2004. Becoming interculturally competent. Toward multiculturalism: A reader in multicultural education/Intercultural Resource Corporation

    2004

  5. [5]

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. 2024. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning

    2024

  6. [6]

    Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2024 a . Culturalteaming: Ai-assisted interactive red-teaming for challenging llms'(lack of) multicultural knowledge. arXiv preprint arXiv:2404.06664

    Pith/arXiv arXiv 2024

  7. [7]

    Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. 2024 b . Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. arXiv preprint arXiv:2410.02677

    Pith/arXiv arXiv 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    Pith/arXiv arXiv 2024

  9. [9]

    P Christopher Earley and Soon Ang. 2003. Cultural intelligence: Individual interactions across cultures

    2003

  10. [10]

    P Christopher Earley and Elaine Mosakowski. 2004. Cultural intelligence. Harvard business review, 82(10):139--146

    2004

  11. [11]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. Eli5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558--3567

    2019

  12. [12]

    Zeyu Gan and Yong Liu. 2024. Towards a theoretical understanding of synthetic data in llm post-training: A reverse-bottleneck perspective. arXiv preprint arXiv:2410.01720

    Pith/arXiv arXiv 2024

  13. [13]

    Jie Gao, Yuchen Guo, Gionnieve Lim, Tianqin Zhang, Zheng Zhang, Toby Jia-Jun Li, and Simon Tangi Perrault. 2024. Collabcoder: a lower-barrier, rigorous workflow for inductive collaborative qualitative analysis with large language models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1--29

    2024

  14. [14]

    Daniel Goleman. 1998. Working with emotional intelligence. NY: Bantam Books

    1998

  15. [15]

    Md Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2024. Nativqa: Multilingual culturally-aligned natural query for llms. arXiv preprint arXiv:2407.09823

    Pith/arXiv arXiv 2024

  16. [16]

    Shreya Havaldar, Bhumika Singhal, Sunny Rai, Langchen Liu, Sharath Chandra Guntuku, and Lyle Ungar. 2023. Multilingual language models are not multicultural: A case study in emotion. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 202--214

    2023

  17. [17]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

    2021

  18. [18]

    Jong Youl Hong. 2023. Multicultural society and intercultural citizens. In Multiculturalism and Interculturalism-Managing Diversity in Cross-Cultural Environment. IntechOpen

    2023

  19. [19]

    Yizheng Huang and Jimmy Huang. 2024. A survey on retrieval-augmented text generation for large language models. arXiv preprint arXiv:2404.10981

    Pith/arXiv arXiv 2024

  20. [20]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys

    2023

  21. [21]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    Pith/arXiv arXiv 2024

  22. [22]

    u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \

    Enkelejda Kasneci, Kathrin Se ler, Stefan K \"u chemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G \"u nnemann, Eyke H \"u llermeier, et al. 2023. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274

    2023

  23. [23]

    Hannah Rose Kirk, Alexander Whitefield, Paul R \"o ttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv pr...

    Pith/arXiv arXiv 2024

  24. [24]

    Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024. Culturepark: Boosting cross-cultural understanding in large language models. arXiv preprint arXiv:2405.15145

    Pith/arXiv arXiv 2024

  25. [25]

    Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024 a . Are multilingual llms culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016--2039

    2024

  26. [26]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024 b . Best practices and lessons learned on synthetic data for language models. arXiv preprint arXiv:2404.07503

    Pith/arXiv arXiv 2024

  27. [27]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511--2522

    2023

  28. [28]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. On llms-driven synthetic data generation, curation, and evaluation: A survey. In Findings of the Association for Computational Linguistics ACL 2024, pages 11065--11082

    2024

  29. [29]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. Large language models: A survey. arXiv preprint arXiv:2402.06196

    Pith/arXiv arXiv 2024

  30. [30]

    Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948

    Pith/arXiv arXiv 2024

  31. [31]

    Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2024. Survey of cultural awareness in language models: Text and beyond. arXiv preprint arXiv:2411.00860

    Pith/arXiv arXiv 2024

  32. [32]

    Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can llm generate culturally relevant commonsense qa data? case study in indonesian and sundanese. arXiv preprint arXiv:2402.17302

    Pith/arXiv arXiv 2024

  33. [33]

    Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464

    Pith/arXiv arXiv 2024

  34. [34]

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rog \'e rio Abreu de Paula, Diyi Yang, et al. 2024. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238

    Pith/arXiv arXiv 2024

  35. [35]

    Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Xiaowen Dong, Yanfeng Wang, and Siheng Chen. 2024. Synthesizing post-training data for llms through multi-agent simulation. arXiv preprint arXiv:2410.14251

    Pith/arXiv arXiv 2024

  36. [36]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    Pith/arXiv arXiv 2024

  37. [37]

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning

    2024

  38. [38]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  39. [39]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  40. [40]

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, et al. 2024. Aflow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762

    Pith/arXiv arXiv 2024

  41. [41]

    Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, and Bin Cui. 2024. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402.19473

    Pith/arXiv arXiv 2024

  42. [42]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  43. [43]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  44. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...