Toward Responsible and Epistemically Grounded Multilingual LLMs for Computational Social Science and Humanities

Wajdi Zaghouani

arxiv: 2606.00596 · v1 · pith:6ZZ7SXMFnew · submitted 2026-05-30 · 💻 cs.CL

Toward Responsible and Epistemically Grounded Multilingual LLMs for Computational Social Science and Humanities

Wajdi Zaghouani This is my paper

Pith reviewed 2026-06-28 19:17 UTC · model grok-4.3

classification 💻 cs.CL

keywords multilingual LLMscomputational social scienceevaluation frameworkcultural alignmentreasoning faithfulnesshermeneuticsinterpretive researchcross-lingual stability

0 comments

The pith

Multilingual LLMs can be evaluated as hermeneutic instruments using metrics for cultural alignment, cross-lingual stability, and reasoning faithfulness in social science and humanities tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that existing NLP benchmarks overlook the interpretive and culturally situated aspects of meaning production when large language models process text across languages. It proposes treating these models as active hermeneutic instruments that shape understanding in social sciences and humanities research. Drawing on philosophy of technology and related traditions, the work constructs a framework that turns those principles into concrete evaluation criteria. The framework includes an experimental protocol, three core metrics, and transparency rules designed specifically for interpretive work. An example applies the protocol to multilingual political discourse analysis to show how the approach supports responsible integration into computational social science.

Core claim

The paper establishes a theoretically grounded evaluation framework that reconceptualizes multilingual reasoning LLMs as hermeneutic instruments and supplies an experimental protocol with operationalized metrics for cultural alignment, cross-lingual stability, and reasoning faithfulness plus transparency requirements tailored to interpretive SSH tasks.

What carries the argument

The evaluation framework that operationalizes hermeneutic concepts into metrics and an experimental protocol for assessing how LLMs mediate meaning across linguistic and cultural contexts.

If this is right

Researchers gain a protocol they can apply directly to multilingual political discourse or similar interpretive tasks.
Evaluations of LLMs for SSH work must now include checks for cultural alignment and reasoning faithfulness in addition to standard accuracy measures.
Transparency requirements become mandatory for any model used in cross-lingual interpretive research.
Computational social science infrastructures can incorporate the framework to guide model selection and auditing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The framework could be extended to test whether the same metrics hold for historical or literary texts beyond political discourse.
Funding or publication guidelines for AI-assisted SSH projects might eventually require documentation of cultural alignment scores.
Neighboring fields such as digital history could adapt the protocol to evaluate models on multilingual archival materials.

Load-bearing premise

Hermeneutic and philosophical concepts about meaning and technology can be converted into quantitative metrics and protocols without losing or distorting their interpretive character.

What would settle it

Human experts in a target SSH domain judge model outputs on an interpretive task as culturally misaligned or unfaithful while the framework's metrics classify the same outputs as aligned and stable.

read the original abstract

Large language models have rapidly evolved in multilingual competence and reasoning capacity, enabling their integration into Social Sciences and Humanities research workflows. Yet existing evaluation paradigms remain anchored in task-based NLP benchmarks and fail to address interpretive validity, cultural situatedness, and epistemic mediation. This paper reconceptualizes multilingual reasoning LLMs as hermeneutic instruments that actively structure meaning production across linguistic and cultural contexts. Drawing on hermeneutics, philosophy of technology, science and technology studies, multilingual NLP research, and computational social science methodology, we develop a theoretically grounded framework for evaluating multilingual reasoning in Social Sciences and Humanities (SSH) research. We articulate a rigorous experimental protocol with operationalized metrics for cultural alignment, cross-lingual stability, and reasoning faithfulness, along with transparency requirements tailored to interpretive research tasks. We illustrate the framework through a concrete application scenario involving multilingual political discourse analysis. The paper contributes a conceptual and methodological foundation for responsible integration of multilingual reasoning LLMs into computational social science infrastructures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a position paper proposing a hermeneutics-based evaluation framework for multilingual LLMs in SSH but delivers only high-level synthesis without tested metrics or data.

read the letter

The core of this paper is a call to treat multilingual LLMs as hermeneutic tools in computational social science and humanities work, with a sketched protocol for checking cultural alignment, cross-lingual stability, and reasoning faithfulness. It draws on existing literature from philosophy, STS, and multilingual NLP to argue for transparency requirements suited to interpretive tasks, and it includes one illustrative scenario on political discourse analysis.

What stands out is the attempt to connect hermeneutic ideas to practical LLM use in SSH. The authors correctly note that standard NLP benchmarks miss cultural situatedness and epistemic mediation, and the scenario gives readers a concrete way to picture where those gaps matter.

The weaknesses are straightforward. The metrics are defined inside the proposed framework itself, with no external benchmarks, prior instruments, or worked numerical examples to show they can be applied or that they capture what is claimed. There are no derivations, datasets, or results that would let someone replicate or falsify the approach. The circularity the reader flagged is real: success is judged against criteria invented for this paper.

This is aimed at researchers already working on responsible AI in the social sciences who want language for discussing interpretive risks. It does not contain enough substance for someone looking for new methods or evidence. A serious referee would likely ask for at least one implemented case with measurable outcomes before considering it further. I would not bring it to a reading group or cite it, and I would desk-reject rather than send to peer review in its current form.

Referee Report

2 major / 1 minor

Summary. The manuscript proposes a theoretically grounded framework for evaluating multilingual LLMs as hermeneutic instruments in SSH research. Drawing on hermeneutics, philosophy of technology, and related fields, it articulates an experimental protocol with operationalized metrics for cultural alignment, cross-lingual stability, and reasoning faithfulness, plus tailored transparency requirements. The framework is illustrated via a concrete application scenario in multilingual political discourse analysis.

Significance. If the operationalized metrics can be shown to preserve interpretive validity, the work would offer a substantive alternative to task-based NLP benchmarks, supporting more epistemically responsible integration of LLMs into computational social science infrastructures.

major comments (2)

[Framework development and metrics description] Abstract and framework articulation: the claim that hermeneutic and STS concepts have been 'directly operationalized' into quantitative metrics for cultural alignment and reasoning faithfulness lacks a concrete derivation or mapping; without an explicit translation step or example computation in the protocol, it is unclear whether the metrics measure the intended epistemic properties or merely restate framework-internal criteria.
[Application scenario] Application scenario: the illustration is described as 'concrete' yet remains at the level of scenario description without reported metric values, baseline comparisons, or external validation against SSH expert judgments; this weakens the demonstration that the protocol is rigorous and non-circular.

minor comments (1)

The abstract would benefit from a brief statement of scope limitations, such as the framework's current status as a conceptual proposal pending empirical testing.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which highlight areas where the manuscript can be strengthened. We address each major comment below and commit to revisions that provide greater explicitness in the framework's operationalization and the application scenario while preserving the paper's conceptual focus.

read point-by-point responses

Referee: Abstract and framework articulation: the claim that hermeneutic and STS concepts have been 'directly operationalized' into quantitative metrics for cultural alignment and reasoning faithfulness lacks a concrete derivation or mapping; without an explicit translation step or example computation in the protocol, it is unclear whether the metrics measure the intended epistemic properties or merely restate framework-internal criteria.

Authors: We agree that an explicit derivation step would improve clarity. The current manuscript describes the metrics at a high level but does not include a dedicated mapping table or worked examples. In revision, we will add a new subsection under the protocol that provides (1) a step-by-step translation from specific hermeneutic/STS concepts to each metric, (2) formal definitions with example computations using sample outputs, and (3) discussion of how the metrics avoid circularity by grounding in external interpretive criteria. This will make the operationalization concrete without altering the framework's theoretical basis. revision: yes
Referee: Application scenario: the illustration is described as 'concrete' yet remains at the level of scenario description without reported metric values, baseline comparisons, or external validation against SSH expert judgments; this weakens the demonstration that the protocol is rigorous and non-circular.

Authors: The scenario is presented as an illustrative walkthrough of protocol application rather than an empirical evaluation, which explains the absence of numerical results or expert validation. We acknowledge this distinction could be clearer. In revision we will (1) relabel the section as 'Illustrative Application' to avoid implying empirical demonstration, (2) insert plausible worked examples of metric values derived from the described discourse data, and (3) add a limitations paragraph noting that full external validation against SSH expert judgments would require a separate empirical study. We maintain that a full empirical test lies outside the scope of this framework paper but will strengthen the illustration accordingly. revision: partial

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper is a conceptual position paper whose central contribution is the explicit construction and articulation of a new evaluation framework, including operationalized metrics for cultural alignment, cross-lingual stability, and reasoning faithfulness. No derivation chain, equations, fitted parameters, or predictions are presented that reduce to inputs by construction. The provided abstract and description contain no self-citations invoked as load-bearing uniqueness theorems, no ansatzes smuggled via prior work, and no renaming of known results. The framework exists by virtue of being specified; this is internally consistent for a methodological proposal and does not trigger any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The claim rests on the untested assumption that hermeneutic and STS concepts translate directly into LLM evaluation metrics; no independent evidence or external benchmarks are referenced for this translation.

axioms (2)

domain assumption Hermeneutic principles from philosophy can be operationalized into quantitative metrics for LLM cultural alignment and reasoning faithfulness without loss of interpretive meaning.
Invoked when the abstract states the framework is 'theoretically grounded' and articulates 'operationalized metrics'.
domain assumption Existing task-based NLP benchmarks are insufficient for interpretive validity in SSH, requiring a new framework.
Stated in the opening of the abstract as the motivation for reconceptualizing LLMs as hermeneutic instruments.

pith-pipeline@v0.9.1-grok · 5690 in / 1435 out tokens · 20703 ms · 2026-06-28T19:17:57.719241+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 4 canonical work pages · 3 internal anchors

[1]

distant reading,

Introduction The development of transformer -based large lan- guage models has reconfigured the technical land- scape of language processing. Models trained on large -scale multilingual corpora demonstrate zero-shot transfer, in-context learning, and chain - of-thought reasoning capacities across a growing number of languages (Brown et al., 2020; Chowd- h...

2020
[2]

technological in- tentionality,

Hermeneutics and Computational Mediation Hermeneutic philosophy offers a foundational lens for understanding interpretation as historically and culturally situated. Gadamer (1975) emphasizes that understanding occurs through a fusion of hori- zons between interpreter and text. Ricoeur (1976) similarly frames interpretation as a dialectical pro- cess that ...

1975
[3]

left-behind

Multilingual Representation and Structural Inequality The multilingual capacity of LLMs is often framed as evidence of inclusivity and global reach. How - ever, empirical research in NLP highlights persis - tent structural asymmetries. Joshi et al. (2020) document severe disparities in resource availability and benchmark inclusion across languages, cat - ...

2020
[4]

Naous et al

Cultural Bias as Epistemic Distortion The abstract concern about cultural homogeniza - tion in multilingual models is supported by a grow- ing body of empirical work that deserves careful attention from the SSH community. Naous et al. (2024) introduce CAMeL, a benchmark of naturally occurring prompts and entities contrasting Arab and Western cultures, and...

2024
[5]

Generating intermediate steps appears to enhance accuracy on arithmetic and logical tasks

Reasoning, Explanation, and Epistemic Authority Chain-of-thought prompting has been shown to im- prove reasoning performance in large language models (Wei et al., 2022). Generating intermediate steps appears to enhance accuracy on arithmetic and logical tasks. However, Turpin et al. (2023) demonstrate that reasoning traces may not reli - ably correspond t...

2022
[6]

converse

LLMs in Computational Social Science Before articulating an evaluation framework, it is important to survey how LLMs are currently being deployed in SSH research. The use of LLMs for text annotation has expanded rapidly, with stud- ies demonstrating that GPT-4 can achieve annota- tion accuracy comparable to or exceeding human crowdworkers across multiple ...

2023
[7]

comparable

Methodological Framework with Operationalized Metrics (ESS). Specifically: • KL Divergence: DKL (Ppopulation ||Pmodel) This section outlines an experimental protocol for evaluating multilingual reasoning LLMs in compu- tational social science contexts, with concrete oper- ationalized metrics addressing cultural alignment, cross-lingual stability, and reas...

2018
[8]

economic burden

Illustrative Application: Multilingual Political Discourse Analysis To demonstrate how the proposed framework can be applied in practice, we outline a concrete re - search scenario involving multilingual political fram- ing analysis, a common task in computational social science. Research question. How do news media in Arabic, English, and French frame im...

2018
[9]

Cultural homogenization risk arises when domi- nant language priors shape interpretation across contexts

Epistemic and Ethical Implications The integration of multilingual reasoning LLMs into SSH research introduces epistemic and ethical con- siderations that extend beyond technical evalua - tion. Cultural homogenization risk arises when domi- nant language priors shape interpretation across contexts. Models trained primarily on English data may impose Weste...

2024
[10]

Codebook provenance: Who defined the anno- tation categories? Were native speakers from target cultures involved?
[11]

Cultural stakeholder engagement: How were affected communities consulted in evaluation design?
[12]

Disaggregation requirements: What subgroup analyses are required to detect differential per- formance?
[13]

Model update monitoring : What procedures exist to detect and document model changes over time?
[14]

Interpretive authority: How are model outputs positioned relative to human expert judgment?
[15]

Prompt sensitivity documentation: Were multi- ple prompt formulations tested, and what was the range of variation in results?
[16]

Cultural alignment verification : Were model outputs assessed for systematic cultural bias using established benchmarks or domain - appropriate cultural probes?
[17]

Existing multilingual bench- marks primarily assess factual knowledge retrieval, natural language inference, or commonsense rea- soning

Toward SSH-Specific Benchmarks The framework proposed here highlights a signifi- cant gap in the current evaluation landscape: the absence of benchmarks specifically designed for SSH interpretive tasks. Existing multilingual bench- marks primarily assess factual knowledge retrieval, natural language inference, or commonsense rea- soning. While these capab...

2019
[18]

and the Aya initiative ( Singh et al. , 2024; Üstün et al., 2024) should inform how native speak- ers and SSH scholars from underrepresented lan- guage communities participate in benchmark de - velopment as researchers rather than merely as annotators

2024
[19]

We address this by proposing a tiered implementation approach

Practical Guidance for Resource-Constrained Settings A realistic concern, raised in the evaluation of this framework, is that the full protocol may be infeasi- ble for researchers lacking access to large multi - lingual annotator pools, validated cultural survey data, or extensive computational resources. We address this by proposing a tiered implementati...

2025
[20]

We recommend that SSH researchers adopt Data Statements ( Bender and Friedman, 2018) and Model Cards (Mitchell et al.,

Implications for the SSH Research Agenda Realizing the potential of multilingual reasoning LLMs for SSH research requires developing shared infrastructure for documenting model configura - tions, logging experimental procedures, and archiv- ing annotation materials. We recommend that SSH researchers adopt Data Statements ( Bender and Friedman, 2018) and M...

2018
[21]

The field would benefit from establishing multilin- gual LLM evaluation consortia that pool expertise across language communities and SSH disciplines

as minimum documentation standards, ex - tended with the epistemic risk register proposed above. The field would benefit from establishing multilin- gual LLM evaluation consortia that pool expertise across language communities and SSH disciplines. Such consortia could maintain living benchmarks that evolve alongside model capabilities, conduct regular cro...
[22]

Conclusion Multilingual reasoning LLMs function as hermeneu- tic instruments that mediate interpretation in com- putational social science. Recognizing this role re- quires moving beyond benchmark accuracy toward epistemically grounded evaluation frameworks that attend to cultural situatedness, reasoning faithful - ness, and the structural inequalities em...
[23]

The paper is primarily theoretical

Limitations This position paper presents a conceptual and methodological framework for evaluating multilin - gual reasoning large language models in Social Sciences and Humanities research. The paper is primarily theoretical. The suggested measures for cultural alignment, cross-lingual stability, and rea- soning faithfulness have not been validated throug...
[24]

Ethical Considerations Integrating multilingual reasoning LLMs into SSH research raises ethical concerns beyond epistemic issues. A primary risk is cultural homogenization from English -dominant training data, which per - petuates Western normative interpretations and marginalizes non-Western epistemologies, indige- nous knowledge, and minority voices. Th...

2020
[25]

K., Bartels, D

Bibliographical References Abdurahman, S., Salkhordeh Ziabari, A., Moore, A. K., Bartels, D. M., and Dehghani, M. 2025. A Primer for Evaluating Large Language Models in Social-Science Research. Advances in Methods and Practices in Psychological Science, 8(2). Adelani, D. I., Abbott, J., Neubig, G., et al. 2021. MasakhaNER: Named entity recognition for Afr...

2025
[26]

On the Opportunities and Risks of Foundation Models

On the opportunities and risks of founda- tion models. arXiv:2108.07258. Brown, T. B., Mann, B., Ryder, N., et al. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Sys - tems, 33. Carlson, N. A. and Burbano, V. 2025. The use of LLMs to annotate data in management research: Foundational guidelines and warnings. St...

work page internal anchor Pith review Pith/arXiv arXiv 2020
[27]

A Survey on LLM-as-a-Judge

XNLI: Evaluating cross -lingual sentence representations. In Proceedings of EMNLP 2018, 2475–2485. Egami, N., Hinck, M., Stewart, B., and Wei, H. 2024. Using large language model annotations for the social sciences: A general framework with appli- cations. Working paper. Entman, R. M. 1993. Framing: Toward clarification of a fractured paradigm. Journal of...

work page internal anchor Pith review Pith/arXiv arXiv 2018
[28]

Measuring Faithfulness in Chain-of-Thought Reasoning

Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702. Latour, B. 1987. Science in Action. Harvard Univer- sity Press. Li, C., Chen, M., Wang, J., Sitaram, S., and Xie, X

work page internal anchor Pith review Pith/arXiv arXiv 1987
[29]

In Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

CultureLLM: Incorporating cultural differ - ences into large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). Liang, P., Bommasani, R., Lee, T., et al. 2023. Holis- tic evaluation of language models. Transactions on Machine Learning Research. Linegar, M., Kocielnik, R., and Alvarez, R. M. 2023. Large language models...

2024
[30]

arXiv preprint arXiv:2003.11529 , year=

MBBQ: A dataset for cross -lingual com- parison of stereotypes in generative LLMs. In Proceedings of COLM 2024. Ollion, E., Shen, R., Macanovic, A., and Chatelain, A. 2023. ChatGPT for text annotation? Mind the hype! SocArXiv. Orife, I., Kreutzer, J., Sibanda, B., et al. 2020. Masakhane – Machine translation for Africa. arXiv:2003.11529. Rathje, S., Mirea...

work page arXiv 2024

[1] [1]

distant reading,

Introduction The development of transformer -based large lan- guage models has reconfigured the technical land- scape of language processing. Models trained on large -scale multilingual corpora demonstrate zero-shot transfer, in-context learning, and chain - of-thought reasoning capacities across a growing number of languages (Brown et al., 2020; Chowd- h...

2020

[2] [2]

technological in- tentionality,

Hermeneutics and Computational Mediation Hermeneutic philosophy offers a foundational lens for understanding interpretation as historically and culturally situated. Gadamer (1975) emphasizes that understanding occurs through a fusion of hori- zons between interpreter and text. Ricoeur (1976) similarly frames interpretation as a dialectical pro- cess that ...

1975

[3] [3]

left-behind

Multilingual Representation and Structural Inequality The multilingual capacity of LLMs is often framed as evidence of inclusivity and global reach. How - ever, empirical research in NLP highlights persis - tent structural asymmetries. Joshi et al. (2020) document severe disparities in resource availability and benchmark inclusion across languages, cat - ...

2020

[4] [4]

Naous et al

Cultural Bias as Epistemic Distortion The abstract concern about cultural homogeniza - tion in multilingual models is supported by a grow- ing body of empirical work that deserves careful attention from the SSH community. Naous et al. (2024) introduce CAMeL, a benchmark of naturally occurring prompts and entities contrasting Arab and Western cultures, and...

2024

[5] [5]

Generating intermediate steps appears to enhance accuracy on arithmetic and logical tasks

Reasoning, Explanation, and Epistemic Authority Chain-of-thought prompting has been shown to im- prove reasoning performance in large language models (Wei et al., 2022). Generating intermediate steps appears to enhance accuracy on arithmetic and logical tasks. However, Turpin et al. (2023) demonstrate that reasoning traces may not reli - ably correspond t...

2022

[6] [6]

converse

LLMs in Computational Social Science Before articulating an evaluation framework, it is important to survey how LLMs are currently being deployed in SSH research. The use of LLMs for text annotation has expanded rapidly, with stud- ies demonstrating that GPT-4 can achieve annota- tion accuracy comparable to or exceeding human crowdworkers across multiple ...

2023

[7] [7]

comparable

Methodological Framework with Operationalized Metrics (ESS). Specifically: • KL Divergence: DKL (Ppopulation ||Pmodel) This section outlines an experimental protocol for evaluating multilingual reasoning LLMs in compu- tational social science contexts, with concrete oper- ationalized metrics addressing cultural alignment, cross-lingual stability, and reas...

2018

[8] [8]

economic burden

Illustrative Application: Multilingual Political Discourse Analysis To demonstrate how the proposed framework can be applied in practice, we outline a concrete re - search scenario involving multilingual political fram- ing analysis, a common task in computational social science. Research question. How do news media in Arabic, English, and French frame im...

2018

[9] [9]

Cultural homogenization risk arises when domi- nant language priors shape interpretation across contexts

Epistemic and Ethical Implications The integration of multilingual reasoning LLMs into SSH research introduces epistemic and ethical con- siderations that extend beyond technical evalua - tion. Cultural homogenization risk arises when domi- nant language priors shape interpretation across contexts. Models trained primarily on English data may impose Weste...

2024

[10] [10]

Codebook provenance: Who defined the anno- tation categories? Were native speakers from target cultures involved?

[11] [11]

Cultural stakeholder engagement: How were affected communities consulted in evaluation design?

[12] [12]

Disaggregation requirements: What subgroup analyses are required to detect differential per- formance?

[13] [13]

Model update monitoring : What procedures exist to detect and document model changes over time?

[14] [14]

Interpretive authority: How are model outputs positioned relative to human expert judgment?

[15] [15]

Prompt sensitivity documentation: Were multi- ple prompt formulations tested, and what was the range of variation in results?

[16] [16]

Cultural alignment verification : Were model outputs assessed for systematic cultural bias using established benchmarks or domain - appropriate cultural probes?

[17] [17]

Existing multilingual bench- marks primarily assess factual knowledge retrieval, natural language inference, or commonsense rea- soning

Toward SSH-Specific Benchmarks The framework proposed here highlights a signifi- cant gap in the current evaluation landscape: the absence of benchmarks specifically designed for SSH interpretive tasks. Existing multilingual bench- marks primarily assess factual knowledge retrieval, natural language inference, or commonsense rea- soning. While these capab...

2019

[18] [18]

and the Aya initiative ( Singh et al. , 2024; Üstün et al., 2024) should inform how native speak- ers and SSH scholars from underrepresented lan- guage communities participate in benchmark de - velopment as researchers rather than merely as annotators

2024

[19] [19]

We address this by proposing a tiered implementation approach

Practical Guidance for Resource-Constrained Settings A realistic concern, raised in the evaluation of this framework, is that the full protocol may be infeasi- ble for researchers lacking access to large multi - lingual annotator pools, validated cultural survey data, or extensive computational resources. We address this by proposing a tiered implementati...

2025

[20] [20]

We recommend that SSH researchers adopt Data Statements ( Bender and Friedman, 2018) and Model Cards (Mitchell et al.,

Implications for the SSH Research Agenda Realizing the potential of multilingual reasoning LLMs for SSH research requires developing shared infrastructure for documenting model configura - tions, logging experimental procedures, and archiv- ing annotation materials. We recommend that SSH researchers adopt Data Statements ( Bender and Friedman, 2018) and M...

2018

[21] [21]

The field would benefit from establishing multilin- gual LLM evaluation consortia that pool expertise across language communities and SSH disciplines

as minimum documentation standards, ex - tended with the epistemic risk register proposed above. The field would benefit from establishing multilin- gual LLM evaluation consortia that pool expertise across language communities and SSH disciplines. Such consortia could maintain living benchmarks that evolve alongside model capabilities, conduct regular cro...

[22] [22]

Conclusion Multilingual reasoning LLMs function as hermeneu- tic instruments that mediate interpretation in com- putational social science. Recognizing this role re- quires moving beyond benchmark accuracy toward epistemically grounded evaluation frameworks that attend to cultural situatedness, reasoning faithful - ness, and the structural inequalities em...

[23] [23]

The paper is primarily theoretical

Limitations This position paper presents a conceptual and methodological framework for evaluating multilin - gual reasoning large language models in Social Sciences and Humanities research. The paper is primarily theoretical. The suggested measures for cultural alignment, cross-lingual stability, and rea- soning faithfulness have not been validated throug...

[24] [24]

Ethical Considerations Integrating multilingual reasoning LLMs into SSH research raises ethical concerns beyond epistemic issues. A primary risk is cultural homogenization from English -dominant training data, which per - petuates Western normative interpretations and marginalizes non-Western epistemologies, indige- nous knowledge, and minority voices. Th...

2020

[25] [25]

K., Bartels, D

Bibliographical References Abdurahman, S., Salkhordeh Ziabari, A., Moore, A. K., Bartels, D. M., and Dehghani, M. 2025. A Primer for Evaluating Large Language Models in Social-Science Research. Advances in Methods and Practices in Psychological Science, 8(2). Adelani, D. I., Abbott, J., Neubig, G., et al. 2021. MasakhaNER: Named entity recognition for Afr...

2025

[26] [26]

On the Opportunities and Risks of Foundation Models

On the opportunities and risks of founda- tion models. arXiv:2108.07258. Brown, T. B., Mann, B., Ryder, N., et al. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Sys - tems, 33. Carlson, N. A. and Burbano, V. 2025. The use of LLMs to annotate data in management research: Foundational guidelines and warnings. St...

work page internal anchor Pith review Pith/arXiv arXiv 2020

[27] [27]

A Survey on LLM-as-a-Judge

XNLI: Evaluating cross -lingual sentence representations. In Proceedings of EMNLP 2018, 2475–2485. Egami, N., Hinck, M., Stewart, B., and Wei, H. 2024. Using large language model annotations for the social sciences: A general framework with appli- cations. Working paper. Entman, R. M. 1993. Framing: Toward clarification of a fractured paradigm. Journal of...

work page internal anchor Pith review Pith/arXiv arXiv 2018

[28] [28]

Measuring Faithfulness in Chain-of-Thought Reasoning

Measuring faithfulness in chain-of-thought reasoning. arXiv:2307.13702. Latour, B. 1987. Science in Action. Harvard Univer- sity Press. Li, C., Chen, M., Wang, J., Sitaram, S., and Xie, X

work page internal anchor Pith review Pith/arXiv arXiv 1987

[29] [29]

In Advances in Neural Information Processing Systems 37 (NeurIPS 2024)

CultureLLM: Incorporating cultural differ - ences into large language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). Liang, P., Bommasani, R., Lee, T., et al. 2023. Holis- tic evaluation of language models. Transactions on Machine Learning Research. Linegar, M., Kocielnik, R., and Alvarez, R. M. 2023. Large language models...

2024

[30] [30]

arXiv preprint arXiv:2003.11529 , year=

MBBQ: A dataset for cross -lingual com- parison of stereotypes in generative LLMs. In Proceedings of COLM 2024. Ollion, E., Shen, R., Macanovic, A., and Chatelain, A. 2023. ChatGPT for text annotation? Mind the hype! SocArXiv. Orife, I., Kreutzer, J., Sibanda, B., et al. 2020. Masakhane – Machine translation for Africa. arXiv:2003.11529. Rathje, S., Mirea...

work page arXiv 2024