Pith. sign in

REVIEW 4 major objections 4 minor 58 references

The paper argues that when alignment training conflicts with a cultural value, models suppress rather than erase it: refusal text conceals high-confidence internal token probabilities, a pattern it calls 'logit leakage'.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:47 UTC pith:OOCE42C5

load-bearing objection The body is a credible MENA-values benchmark paper; the abstract advertises a different, larger study, and the 'logit leakage' phenomenon is not yet backed by the right controls. the 4 major comments →

arxiv 2510.13154 v2 pith:OOCE42C5 submitted 2025-10-15 cs.CL

The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs

Pardis Sadat Zahraei , Gokhan Tur , Dilek Hakkani-T\"ur , Ehsaneddin Asgari This is my paper
classification cs.CL
keywords MENA Values benchmarkcultural alignmentlogit leakagerefusal behaviorcross-lingual value shiftreasoning-induced degradationmultilingual biassurvey-based evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces MENA Values, a benchmark of 864 multiple-choice questions built from the World Values Survey Wave 7 and the Arab Opinion Index 2022, with population-level response distributions for 16 MENA countries. Using seven LLMs and a grid of three perspective framings (neutral, persona, cultural observer), two languages (English vs Arabic/Persian/Turkish), and two reasoning conditions (zero-shot vs with-reasoning), it tries to establish that alignment training suppresses cultural knowledge rather than removing it. The central evidence is 'logit leakage': when a model refuses a sensitive question ('I cannot have personal beliefs'), its normalized token probabilities often assign more than 75% probability mass to one answer option, with leakage rates ranging from 6.95% to 47.50% across models in the with-reasoning condition. It also reports 'reasoning-induced degradation' (NVAS falling 3.52–6.96 points for Mistral, Llama-3.1, and Fanar) and 'cross-lingual value shifts' in which the same question gets different answers in different languages, with native-language prompts collapsing countries into language-based clusters. The abstract frames the result as an 'alignment veto' with a 37.6% safety tax, a 19.8% alignment gap, and a sparse-autoencoder/DPO mechanism; those numbers and the mechanistic claim do not appear in the body, where the empirical support is the logit-leakage and degradation evidence.

Core claim

In the body, the central empirical discovery is 'logit leakage': models that refuse culturally sensitive multiple-choice questions often leave a clear trace of a hidden preference in token-level probabilities. Defining 'strong internal conviction' as >75% of maximum normalized probability, the paper reports refusal-with-conviction rates of 5.86–33.65% (zero-shot) and 6.95–47.50% (with reasoning) across open models. The authors argue safety training teaches models to hide biases rather than resolve them, and distinguish suppression failures (accurate internal distribution, blocked output) from representational-bias failures (encoding diverges from human values). Reasoning prompts usually lowe

What carries the argument

The load-bearing object is the normalized token-level probability distribution over answer options, extracted from the first tokens of the model's generation and compared to human survey response distributions via KL divergence. This is the instrument that makes 'logit leakage' visible: a surface refusal with >75% probability mass on one option is read as a suppressed internal preference. The same machinery supports the paper's contrast between suppression and representational bias, since a distribution can match human data (suppression) or diverge from it (bias).

Load-bearing premise

The load-bearing assumption is that normalized token probabilities over answer options reflect the model's true internal preference, so a refusal paired with one high-probability option reveals suppressed cultural knowledge; the paper itself concedes in the Discussion that interpreting token probabilities as 'beliefs' is contested, and if logits instead reflect surface-linguistic likelihoods or prompt artifacts, logit leakage loses its meaning.

What would settle it

For each refusal item, run a forced-choice version with the same options but no identity framing, and compare the leaked high-probability option against the human modal response; if the leaked option does not beat chance at predicting the human answer, the 'suppressed knowledge' reading fails. Conversely, if changing only the refusal wording — e.g., instructing the model to 'respond as a neutral sociologist' — shifts the leaked distribution, then the internal state is prompt-bound rather than a stable hidden belief.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Output-only safety evaluation will systematically underestimate bias if refusals can coexist with high-confidence hidden choices.
  • Reasoning prompts are not a neutral diagnostic; they can move a model's measured cultural position, so chain-of-thought deployments need cultural-side checks.
  • Multilingual deployment does not guarantee cultural fidelity: native-language prompts can erase country-level distinctions rather than restore them.
  • Suppression and representational bias imply different fixes, since the first is an output-gate problem and the second an encoding problem.
  • The benchmark's question set and human distributions give other regions a reusable template for measuring where models diverge from documented public opinion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that logit leakage should generalize beyond cultural values to any domain where alignment targets conflict with pretraining knowledge (political opinions, commercial claims). A testable prediction: identical refusal templates on factual questions should not show the same stable high-confidence pattern.
  • Given the paper's own caveat that token probabilities as 'beliefs' is contested, the leaked option may be the most fluent next token rather than a held value; comparing leakage on value items with leakage on neutral trivia under the same refusal template would separate these readings.
  • The body's abstention tables contain the raw material for an equity analysis the abstract promises but the body does not compute: uneven refusal rates across nations could translate into uneven suppression, i.e., some populations' values being more blocked than others.
  • One of the paper's qualitative failure modes (safety-induced self-censorship) suggests a cheap intervention: altering reasoning prompts to suppress hedging may recover measured alignment without weight updates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MENA Values, a benchmark of 864 multiple-choice questions derived from the World Values Survey Wave 7 and the Arab Opinion Index 2022, covering 16 MENA countries. Seven LLMs are evaluated under three perspective framings (neutral, persona, observer), two languages (English and native), and two reasoning conditions (zero-shot, with-reasoning). The full-text abstract and body report three phenomena: Cross-Lingual Value Shifts, Reasoning-Induced Degradation, and Logit Leakage, with additional PCA-based analyses of linguistic determinism and cultural identity crisis. The submitted abstract, however, claims a broader 'alignment veto' with a 37.6% safety tax, a 19.8% alignment-quality gap, and sparse-autoencoder/DPO-stage evidence; these claims do not appear in the full text.

Significance. If the benchmark and its findings hold, the paper provides a valuable diagnostic resource for an under-represented region, with a large human-survey-derived ground truth, bootstrapped confidence intervals, and a plan to release code and data. The identification of reasoning-induced degradation and language-driven clustering are potentially useful empirical observations. However, the headline claims in the submitted abstract are unsupported by the manuscript body, and the central novel phenomenon, logit leakage, rests on an unvalidated interpretation of token probabilities. The undefined distance function D in the consistency metrics makes several reported scores non-recomputable. These issues substantially undermine the paper's core contributions as currently presented.

major comments (4)
  1. [Abstract vs. Full Text] The submitted abstract claims a 'safety tax' of 37.6%, a 19.8% alignment-quality gap, and sparse-autoencoder/DPO-stage evidence for a suppression mechanism. None of these appear in the body (Sections 1–8) or appendices: there is no definition of 'safety tax,' no SAE analysis, no DPO-stage feature, and no 37.6%/19.8% numbers. The paper's title and headline concept ('alignment veto') therefore rest on results that are not in the manuscript. The full-text abstract instead describes logit leakage, reasoning degradation, and cross-lingual shifts, which are different claims.
  2. [§5.1.2, Eqs. (2)–(4)] FCS, CLCS, and SPD all rely on a 'normalized distance function D(v1, v2)' that is never defined. Without a specification of D (e.g., total variation distance, normalized L1, or Jensen–Shannon distance), these scores in Table 1 cannot be computed or independently verified. This is not a cosmetic gap: the consistency metrics are used to support the cross-lingual and framing claims.
  3. [§5.2.1, Table 2] Logit leakage is operationalized purely as a threshold on normalized token log-probabilities (>75% of the maximum), with no control for option-position bias, label tokenization, or refusal-template continuation. The paper itself cites Zheng et al. (2024) showing LLMs are not robust multiple-choice selectors, yet no option-order permutation or forced-choice validation is reported. The examples in Appendix C show the 'leaked' option is often simply the first or last label. Without such controls, leakage rates (e.g., 47.50% for Fanar) may reflect surface artifacts rather than suppressed cultural knowledge. The Discussion concedes that 'the interpretation of token probabilities as beliefs remains contested'; the paper does not resolve it.
  4. [Table 2 and Abstract] The abstract claims that 'at the moment of refusal, a model's internal logit distribution correlates with human survey data more strongly than its freely generated answers.' No such correlation or statistical test is reported in the full text; Table 2 lists leakage rates without confidence intervals or significance tests. The claim is therefore unsupported.
minor comments (4)
  1. [§4] The phrase '322 evaluation matrix' should read '3×2×2 evaluation matrix' (three perspectives, two languages, two reasoning conditions).
  2. [Appendix references] Cross-references to appendices are inconsistent: 'Appendix 2.1', 'Appendix 6.3 and 6.5', 'Appendix F', etc., do not correspond to the labeled appendix sections. Please standardize.
  3. [§3.2 and Figure 4] The text states that Jensen-Shannon Divergence analysis shows distributional similarity scores >0.95 between country pairs, but Figure 4(a)–(d) display similarity scores in the range 0.1–0.4 for individual thematic categories. The >0.95 value appears to refer only to the overall aggregate panel (e). Clarify this distinction to avoid misleading readers.
  4. [Throughout] Model naming is inconsistent: 'ALLAM' vs 'ALLaM', 'AY A' vs 'Aya', 'GTP-4o-mini' vs 'GPT-4o-mini'. Please standardize.

Circularity Check

0 steps flagged

No significant circularity: the benchmark compares LLM outputs to external human-survey ground truth, with no fitted parameter or load-bearing self-citation.

full rationale

The paper's derivation chain is an external benchmark evaluation, not a fitted or self-referential prediction. Ground truth comes from WVS-7 and AOI-2022 human survey distributions, and model outputs and token probabilities are measured against those external responses; no parameter is fitted to the data in order to produce the reported NVAS, CLCS, FCS, SPD, or leakage rates. The 75% 'strong internal conviction' threshold in Section 5.2.1 is an explicit operational definition, not a quantity fitted to maximize findings, and the measured fact 'refusal with high normalized log-probability' is not constructed from the paper's conclusions. The interpretation of that probability as a 'hidden belief' is flagged by the authors themselves in the Discussion ('the interpretation of token probabilities as “beliefs” remains contested') and in the Limitations; that is an assumption/validity concern, not circularity. The only self-touch is that one co-author is also on the Fanar model team, but Fanar is evaluated as an external system and the Fanar citation is not used to justify any theoretical claim. The abstract's 37.6% safety tax and DPO/sparse-autoencoder claims are absent from the full text, which is a reporting gap rather than a circular step. No equation reduces to its own input and no load-bearing result rests on a self-citation chain.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claims rest primarily on survey ground truth and logit-interpretation assumptions. No new physical or conceptual entities are introduced; the 'alignment veto' from the abstract is not developed in the body.

free parameters (1)
  • strong conviction threshold = 0.75 (75% of maximum probability)
    Section 5.2.1 defines a 'strong internal conviction' as normalized log-probability >75% of the maximum; this threshold is chosen by the authors and not derived from data.
axioms (4)
  • domain assumption Logit probabilities over answer tokens reflect latent model preferences/beliefs
    Logit-leakage interpretation requires this; the paper itself notes the interpretation is contested (Discussion).
  • domain assumption WVS-7 and AOI-2022 survey data, after post-stratification, are valid ground truth for country-level cultural values
    Used as benchmark targets; if surveys are unrepresentative, NVAS and related metrics are meaningless.
  • ad hoc to paper The distance function D in FCS/CLCS/SPD is a valid normalized metric
    D is used in Eqs. 2–4 but never defined; the consistency scores are uncomputable without it.
  • domain assumption Persona prompts ('Imagine you are an average [nationality]') activate culturally specific representations rather than stereotypes
    Used to measure persona-based alignment; the paper's own qualitative analysis shows stereotype activation in some cases.

pith-pipeline@v1.3.0-alltime-deepseek · 44082 in / 12835 out tokens · 107379 ms · 2026-08-04T09:47:09.414562+00:00 · methodology

0 comments
read the original abstract

What happens inside a language model when alignment training conflicts with a cultural value it encodes? Across 16 MENA countries, 26 models, and 1.53M human survey responses, we show the answer is suppression, not erasure: at the moment of refusal, a model's internal logit distribution correlates with human survey data more strongly than its freely generated answers. We call this the alignment veto. We distinguish suppression failures (accurate internal distributions blocked at output) from representational bias failures (the encoding itself diverges from human values), and show the two require different interventions. The gate is inequitable: the safety tax reaches 37.6%, with a 19.8% alignment-quality gap between best- and worst-served nations, and native-language prompting widens rather than closes it. Sparse autoencoder analysis corroborated by comparisons across alignment stages identifies a candidate DPO-stage feature mediating suppression in Tulu-3-8B. The gate is real, its costs are unequal, and deciding what it should protect is not a technical question.

Figures

Figures reproduced from arXiv: 2510.13154 by Dilek Hakkani-T\"ur, Ehsaneddin Asgari, Gokhan Tur, Pardis Sadat Zahraei.

Figure 1
Figure 1. Figure 1: Systematic Value Inconsistency in LLMs: A Multi-Dimensional Analysis of Alignment Failures. This figure reveals how LLMs exhibit inconsistencies when responding to value-based questions, demonstrating three critical dimensions of misalignment. Cross-Lingual Value Shift shows how identical questions yield contradictory responses across languages (Arabic vs. English), suggesting cultural bias encoding rather… view at source ↗
Figure 2
Figure 2. Figure 2: Core Dimensions of the MENAVALUES Dataset. The dataset is structured around four major pillars: (1) Social & Cultural Identity, (2) Economic Dimensions, (3) Governance & Political Systems, and (4) Individual Wellbeing & Development. Each category is illustrated with survey questions and average responses from representative MENA countries. Note: The countries shown are illustrative examples; all benchmark … view at source ↗
Figure 3
Figure 3. Figure 3: Principal Component Analysis of WVS-7 countries. [PITH_FULL_IMAGE:figures/full_fig_p028_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distributional similarity heatmaps showing Jensen-Shannon Divergence-based similarity [PITH_FULL_IMAGE:figures/full_fig_p030_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Principal Component Analysis of AOI countries. [PITH_FULL_IMAGE:figures/full_fig_p031_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distributional similarity heatmaps showing Jensen-Shannon Divergence-based similarity [PITH_FULL_IMAGE:figures/full_fig_p032_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of model performance across four evaluation metrics with and without [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: PCA of ALLaM’s cultural representations (English, Observer). The shift in country clusters [PITH_FULL_IMAGE:figures/full_fig_p035_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: PCA of Aya’s cultural representations (English, Observer). The shift in country clusters [PITH_FULL_IMAGE:figures/full_fig_p035_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: PCA of GPT-4’s cultural representations (English, Observer). The shift in country clusters [PITH_FULL_IMAGE:figures/full_fig_p036_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: PCA of Fanar’s cultural representations (English, Observer). The shift in country clusters [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: PCA of Gemini’s cultural representations (English, Observer). The shift in country [PITH_FULL_IMAGE:figures/full_fig_p037_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: PCA of Llama 3.1’s cultural representations (English, Observer). The shift in country [PITH_FULL_IMAGE:figures/full_fig_p037_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: PCA of Mistral’s cultural representations (English, Observer). The shift in country clusters [PITH_FULL_IMAGE:figures/full_fig_p038_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: PCA of ALLaM’s cultural representations using native-language prompts, demonstrating [PITH_FULL_IMAGE:figures/full_fig_p038_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: PCA of Aya’s cultural representations using native-language prompts, demonstrating the [PITH_FULL_IMAGE:figures/full_fig_p039_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: PCA of GPT4’s cultural representations using native-language prompts, demonstrating [PITH_FULL_IMAGE:figures/full_fig_p039_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: PCA of Fanar’s cultural representations using native-language prompts, demonstrating the [PITH_FULL_IMAGE:figures/full_fig_p040_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: PCA of Gemini’s cultural representations using native-language prompts, demonstrating [PITH_FULL_IMAGE:figures/full_fig_p040_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: PCA of Llama3.1’s cultural representations using native-language prompts, demonstrating [PITH_FULL_IMAGE:figures/full_fig_p041_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: PCA of Mistral’s cultural representations using native-language prompts, demonstrating [PITH_FULL_IMAGE:figures/full_fig_p041_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: PCA of ALLaM’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p042_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: PCA of Aya’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p042_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: PCA of GPT-4’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p043_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: PCA of Fanar’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p043_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: PCA of Gemini’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p044_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: PCA of Llama 3.1’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p044_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: PCA of Mistral’s persona-based representations, illustrating the model’s [PITH_FULL_IMAGE:figures/full_fig_p045_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: PCA of ALLaM’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p045_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: PCA of Aya’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p046_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: PCA of GPT-4’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p046_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: PCA of Fanar’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p047_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: PCA of Gemini’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p047_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: PCA of Llama 3.1’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p048_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: PCA of Mistral’s persona-based representations (Native Languages). This confirms the [PITH_FULL_IMAGE:figures/full_fig_p048_35.png] view at source ↗
Figure 36
Figure 36. Figure 36: PCA of ALLaM’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p049_36.png] view at source ↗
Figure 37
Figure 37. Figure 37: PCA of Aya’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p049_37.png] view at source ↗
Figure 38
Figure 38. Figure 38: PCA of GPT-4’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p050_38.png] view at source ↗
Figure 39
Figure 39. Figure 39: PCA of Fanar’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p050_39.png] view at source ↗
Figure 40
Figure 40. Figure 40: PCA of Gemini’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p051_40.png] view at source ↗
Figure 41
Figure 41. Figure 41: PCA of Llama 3.1’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p051_41.png] view at source ↗
Figure 42
Figure 42. Figure 42: PCA of Mistral’s neutral responses, providing direct evidence for [PITH_FULL_IMAGE:figures/full_fig_p052_42.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 2 canonical work pages

  1. [1]

    Persistent anti-muslim bias in large language models, 2021

    Abubakar Abid, Maheen Farooqi, and James Zou. Persistent anti-muslim bias in large language models, 2021. URL https://arxiv.org/abs/2101.05783

    Pith/arXiv arXiv 2021

  2. [2]

    Towards measuring and modeling culture in LLM s: A survey

    Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O ' Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling culture in LLM s: A survey. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Languag...

    doi:10.18653/v1/2024.emnlp-main 2024

  3. [3]

    Investigating cultural alignment of large language models

    Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. Investigating cultural alignment of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 12404--12422, Bangkok, Thailand, August 2024 a . Assoc...

    doi:10.18653/v1/2024.acl-long.671 2024

  4. [4]

    Investigating cultural alignment of large language models, 2024 b

    Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. Investigating cultural alignment of large language models, 2024 b . URL https://arxiv.org/abs/2402.13231

    Pith/arXiv arXiv 2024

  5. [5]

    Arab opinion index 2022

    Arab Center for Research and Policy Studies . Arab opinion index 2022. https://arabindex.dohainstitute.org/EN/Pages/Arab-Opinion-Index-2022.aspx, 2022. Eighth wave of the Arab Opinion Index, based on face-to-face interviews with 33,300 respondents across 14 Arab countries

    2022

  6. [6]

    Probing pre-trained language models for cross-cultural differences in values

    Arnav Arora, Lucie-aim \'e e Kaffee, and Isabelle Augenstein. Probing pre-trained language models for cross-cultural differences in values. In Sunipa Dev, Vinodkumar Prabhakaran, David Ifeoluwa Adelani, Dirk Hovy, and Luciana Benotti (eds.), Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pp.\ 114--130, Dubrovnik, Croati...

    doi:10.18653/v1/2023.c3nlp-1.12 2023

  7. [7]

    Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or abduction: How do LLM s answer multiple-choice questions without the question? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 10308--10330, Bangkok, Thaila...

    doi:10.18653/v1/2024.acl-long.555 2024

  8. [8]

    Assessing cross-cultural alignment between C hat GPT and human societies: An empirical study

    Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. Assessing cross-cultural alignment between C hat GPT and human societies: An empirical study. In Sunipa Dev, Vinodkumar Prabhakaran, David Ifeoluwa Adelani, Dirk Hovy, and Luciana Benotti (eds.), Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP...

    doi:10.18653/v1/2023.c3nlp-1.7 2023

  9. [9]

    Marked personas: Using natural language prompts to measure stereotypes in language models

    Myra Cheng, Esin Durmus, and Dan Jurafsky. Marked personas: Using natural language prompts to measure stereotypes in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1504--1532, Toronto, Canada, July 2023. As...

    doi:10.18653/v1/2023.acl-long.84 2023

  10. [10]

    A nthro S core: A computational linguistic measure of anthropomorphism

    Myra Cheng, Kristina Gligoric, Tiziano Piccardi, and Dan Jurafsky. A nthro S core: A computational linguistic measure of anthropomorphism. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 807--825, St. Julian ' s, Malta, Mar...

    2024

  11. [11]

    Aya expanse: Combining research breakthroughs for a new multilingual frontier, 2024

    John Dang, Shivalika Singh, Daniel D'souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venk...

    Pith/arXiv arXiv 2024

  12. [12]

    Toxicity in chatgpt: Analyzing persona-assigned language models

    Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp.\ 1236--1270, Singapore, December 2023. Association for Computational Li...

    doi:10.18653/v1/2023.findings-emnlp.88 2023

  13. [13]

    S tereo KG : Data-driven knowledge graph construction for cultural knowledge and stereotypes

    Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. S tereo KG : Data-driven knowledge graph construction for cultural knowledge and stereotypes. In Kanika Narang, Aida Mostafazadeh Davani, Lambert Mathias, Bertie Vidgen, and Zeerak Talat (eds.), Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH), pp.\ 67--78, Seattle, Was...

    doi:10.18653/v1/2022.woah-1.7 2022

  14. [15]

    Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language mod...

    Pith/arXiv arXiv 2024

  15. [16]

    Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilingual language models think better in E nglish? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short ...

    doi:10.18653/v1/2024.naacl-short.46 2024

  16. [17]

    Discovering language-neutral sub-networks in multilingual language models

    Negar Foroutan, Mohammadreza Banaei, R \'e mi Lebret, Antoine Bosselut, and Karl Aberer. Discovering language-neutral sub-networks in multilingual language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.\ 7560--7575, Abu Dhabi, United Arab Emirat...

    doi:10.18653/v1/2022.emnlp-main.513 2022

  17. [18]

    Massively multi-cultural knowledge acquisition & lm benchmarking, 2024

    Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & lm benchmarking, 2024. URL https://arxiv.org/abs/2402.09369

    Pith/arXiv arXiv 2024

  18. [19]

    The challenge of value alignment: from fairer algorithms to ai safety, 2021

    Iason Gabriel and Vafa Ghazavi. The challenge of value alignment: from fairer algorithms to ai safety, 2021. URL https://arxiv.org/abs/2101.06060

    Pith/arXiv arXiv 2021

  19. [20]

    Gemini 2.5 flash-lite

    Google Cloud . Gemini 2.5 flash-lite. https://cloud.google.com/vertex-ai/generative-ai/docs/models/gemini/2-5-flash-lite, 2025. Accessed: 2025-08-24

    2025

  20. [21]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    Pith/arXiv arXiv 2024

  21. [22]

    Sociodemographic bias in language models: A survey and forward path

    Vipul Gupta, Pranav Narayanan Venkit, Shomir Wilson, and Rebecca Passonneau. Sociodemographic bias in language models: A survey and forward path. In Agnieszka Fale \'n ska, Christine Basta, Marta Costa-juss \`a , Seraphina Goldfarb-Tarrant, and Debora Nozza (eds.), Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp....

    work page doi:10.18653/v1/2024.gebnlp-1.19 2024

  22. [23]

    World values survey: Round seven – country-pooled datafile version 6.0

    Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, Bi Puranen, et al. World values survey: Round seven – country-pooled datafile version 6.0. https://doi.org/10.14281/18241.24, 2022. Editors

    doi:10.14281/18241.24 2022

  23. [24]

    Aligning ai with shared human values, 2023

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values, 2023. URL https://arxiv.org/abs/2008.02275

    Pith/arXiv arXiv 2023

  24. [25]

    Challenges and strategies in cross-cultural NLP

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders S gaard. Challenges and strategies in cross-cultural NLP . In Smaranda Muresan, Preslav Nakov, and Aline Villavicenci...

    2022

  25. [26]

    Challenges and strategies in cross-cultural NLP

    Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders S gaard. Challenges and strategies in cross-cultural NLP . In Smaranda Muresan, Preslav Nakov, and Aline Villavicenci...

    2022

  26. [27]

    A ce GPT , localizing large language models in A rabic

    Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. A ce GPT , localizing large language models in A rabic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceeding...

    2024

  27. [28]

    Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023. URL https://arxiv.org/abs/2310.11564

    Pith/arXiv arXiv 2023

  28. [29]

    S ee GULL : A stereotype benchmark with broad geo-cultural coverage leveraging generative models

    Akshita Jha, Aida Mostafazadeh Davani, Chandan K Reddy, Shachi Dave, Vinodkumar Prabhakaran, and Sunipa Dev. S ee GULL : A stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    doi:10.18653/v1/2023.acl-long 2023

  29. [30]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxi...

    Pith/arXiv arXiv 2023

  30. [31]

    Personas as a way to model truthfulness in language models

    Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas as a way to model truthfulness in language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 6346--6359, Miami, Florida, USA, November 2024. Association for Computati...

    work page doi:10.18653/v1/2024.emnlp-main.364 2024

  31. [32]

    How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions, 2024

    Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions, 2024. URL https://arxiv.org/abs/2406.14805

    Pith/arXiv arXiv 2024

  32. [33]

    Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback, 2023. URL https://arxiv.org/abs/2303.05453

    Pith/arXiv arXiv 2023

  33. [34]

    Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language...

    Pith/arXiv arXiv 2024

  34. [35]

    Culturellm: Incorporating cultural differences into large language models, 2024 a

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. Culturellm: Incorporating cultural differences into large language models, 2024 a . URL https://arxiv.org/abs/2402.10946

    Pith/arXiv arXiv 2024

  35. [37]

    Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. Can multiple-choice questions really be useful in detecting the abilities of LLM s? In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan...

    2024

  36. [38]

    Towards understanding and mitigating social biases in language models, 2021

    Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models, 2021. URL https://arxiv.org/abs/2106.13219

    Pith/arXiv arXiv 2021

  37. [39]

    SHADES : Towards a multilingual assessment of stereotypes in large language models

    Margaret Mitchell, Giuseppe Attanasio, Ioana Baldini, Miruna Clinciu, Jordan Clive, Pieter Delobelle, Manan Dey, Sil Hamilton, Timm Dill, Jad Doughman, Ritam Dutt, Avijit Ghosh, Jessica Zosa Forde, Carolin Holtermann, Lucie-Aim \'e e Kaffee, Tanmay Laud, Anne Lauscher, Roberto L Lopez-Davila, Maraim Masoud, Nikita Nangia, Anaelia Ovalle, Giada Pistilli, D...

    2025

  38. [40]

    S tereo S et: Measuring stereotypical bias in pretrained language models

    Moin Nadeem, Anna Bethke, and Siva Reddy. S tereo S et: Measuring stereotypical bias in pretrained language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: ...

    doi:10.18653/v1/2021.acl-long.416 2021

  39. [42]

    Ryan, Alan Ritter, and Wei Xu

    Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models, 2024 b . URL https://arxiv.org/abs/2305.14456

    Pith/arXiv arXiv 2024

  40. [43]

    Having beer after prayer? measuring cultural bias in large language models

    Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 16366--16393, Bangkok, Thailand, August 2024 c . Ass...

    doi:10.18653/v1/2024.acl-long.862 2024

  41. [44]

    Gpt-4o mini: advancing cost-efficient intelligence

    OpenAI . Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/, 2025. Accessed: 2025-08-24

    2025

  42. [45]

    Lifting the curse of multilinguality by pre-training modular transformers

    Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational ...

    2022

  43. [46]

    Cross-lingual consistency of factual knowledge in multilingual language models

    Jirui Qi, Raquel Fern \'a ndez, and Arianna Bisazza. Cross-lingual consistency of factual knowledge in multilingual language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 10650--10666, Singapore, December 2023. Association for Computational Linguist...

    doi:10.18653/v1/2023.emnlp-main.658 2023

  44. [47]

    Allam-thinking: Arabic large language model with enhanced reasoning capabilities

    Mohammed Al-Maghrabi Research. Allam-thinking: Arabic large language model with enhanced reasoning capabilities. https://huggingface.co/almaghrabima/ALLaM-Thinking, 2025

    2025

  45. [48]

    Unintended impacts of LLM alignment on global representation

    Michael J Ryan, William Held, and Diyi Yang. Unintended impacts of LLM alignment on global representation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 16121--16140, Bangkok, Thailand, August 2024. Association for Computational...

    doi:10.18653/v1/2024.acl-long.853 2024

  46. [49]

    The language barrier: Dissecting safety challenges of LLM s in multilingual contexts

    Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of LLM s in multilingual contexts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp.\ 2668--2680, B...

    doi:10.18653/v1/2024.findings-acl.156 2024

  47. [50]

    Understanding the capabilities and limitations of large language models for cultural commonsense

    Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihalcea. Understanding the capabilities and limitations of large language models for cultural commonsense. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguist...

    2024

  48. [51]

    Societal biases in language generation: Progress and challenges

    Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. Societal biases in language generation: Progress and challenges. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (...

    doi:10.18653/v1/2021.acl-long.330 2021

  49. [52]

    Culturebank: An online community-driven knowledge base towards culturally aware language technologies, 2024

    Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua yu, Raya Horesh, Rogério Abreu de Paula, and Diyi Yang. Culturebank: An online community-driven knowledge base towards culturally aware language technologies, 2024. URL https://arxiv.org/abs/2404.15238

    Pith/arXiv arXiv 2024

  50. [54]

    Cultural bias and cultural alignment of large language models

    Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3 0 (9), September 2024 b . ISSN 2752-6542. doi:10.1093/pnasnexus/pgae346. URL http://dx.doi.org/10.1093/pnasnexus/pgae346

    doi:10.1093/pnasnexus/pgae346 2024

  51. [55]

    Fanar: An arabic-centric multimodal generative ai platform, 2025

    Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus'ab Husaini, Soon-G...

    Pith/arXiv arXiv 2025

  52. [56]

    S ea E val for multilingual foundation models: From cross-lingual alignment to cultural reasoning

    Bin Wang, Zhengyuan Liu, Xin Huang, Fangkai Jiao, Yang Ding, AiTi Aw, and Nancy Chen. S ea E val for multilingual foundation models: From cross-lingual alignment to cultural reasoning. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

    2024

  53. [57]

    Not all countries celebrate thanksgiving: On the cultural dominance in large language models

    Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    doi:10.18653/v1/2024.acl-long.345 2024

  54. [58]

    Large language models are not robust multiple choice selectors, 2024

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors, 2024. URL https://arxiv.org/abs/2309.03882

    Pith/arXiv arXiv 2024

  55. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  56. [60]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  57. [61]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  58. [62]

    &51͹ b6c#f> g Cd f9 & `L /8OC >`l

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    Pith/arXiv arXiv 2025