Understanding the Impact of Linguistic Realization Choices on LLM Stance with Causal Tracing

Franziska Weeber; Langchen Huang; Sebastian Pad\'o

REVIEW 3 major objections 5 minor 23 references

The paper claims that controlled linguistic rewrites—including meaning-preserving ones—systematically shift LLM political stance judgments, and activation patching localizes the shift to mid-to-late decoder block outputs at the final prompt

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:42 UTC pith:LHCPU5CK

load-bearing objection Useful, honest empirical study; the behavioral claim about constructional stance shifts is probably right, but control validity and missing error bars on patching keep the localization claim conditional. the 3 major comments →

arxiv 2607.20115 v1 pith:LHCPU5CK submitted 2026-07-22 cs.CL

Understanding the Impact of Linguistic Realization Choices on LLM Stance with Causal Tracing

Langchen Huang , Sebastian Pad\'o , Franziska Weeber This is my paper

classification cs.CL

keywords LLM stancelinguistic constructionmeaning-preserving rewritesactivation patchingpolitical stancesemantic invarianceit-cleftsvariance decomposition

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the grammatical form of a policy statement can change a language model's political stance, even when the meaning is preserved. Using four open-weight models and a set of political statements with six controlled rewrite types, it finds that meaning-preserving constructions—active/passive, it-clefts, wh-clefts, and support-verb constructions—do shift agreement judgments, though less than negation and semantic opposites. Activation patching, which replaces internal activations from one phrasing with those from another, shows that mid-to-late decoder layer outputs at the final prompt position are sufficient to restore the original stance, while attention and MLP sublayer patches are weaker. If correct, stance judgments are not fixed by propositional content alone, and syntactic realization is a causal variable that evaluation pipelines must control.

Core claim

Across two model families and four model sizes, stance instability appears under both meaning-reversing edits (explicit negation, semantic opposites) and meaning-preserving syntactic rewrites (active/passive, it-clefts, wh-clefts, support-verb constructions). Variance decomposition attributes more variation to issue content than to phrasing, but phrasing alone moves judgments by roughly half to over one Likert point on average, beyond sampling noise. Activation patching at the final answer position shows that swapping in block-output activations from the original statement restores the original stance distribution most strongly in mid-to-late decoder layers; attention and MLP patches mostly

What carries the argument

The central machinery is a set of six controlled linguistic rewrites—explicit negation, semantic opposites, active/passive, it-clefts, wh-clefts, and support-verb constructions—each either preserving or systematically inverting truth conditions, paired with activation patching. Patching substitutes activations from the original statement into the forward pass of the rewritten statement at three sites per layer (block output, attention sublayer, MLP sublayer) and measures how much the original seven-point Likert distribution is restored, using 1-Wasserstein distance and a normalized restoration score.

Load-bearing premise

The rewrites must actually preserve or invert the statement's meaning as labeled; the paper concedes that small semantic changes may remain despite manual validation, and if clefts or passives introduce presuppositions or content shifts, the conclusion that construction alone drives stance changes would not hold.

What would settle it

Annotate a random sample of the meaning-preserving rewrites for whether they preserve truth conditions and presuppositions; if a large share of it-clefts or passives introduces presuppositions not present in the base (e.g., an existential presupposition of the clefted constituent), then the observed stance shifts could be semantic artifacts, falsifying the claim that constructional choice alone shifts stance.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Meaning-preserving rewrites must be treated as a potential source of measurement error in any LLM-based stance or opinion evaluation.
Larger models are not simply more robust: they flip less often but also retreat toward neutral, so scale alone does not guarantee semantic invariance.
Because the restoration signal concentrates in block outputs at the final prompt position, interventions at those layers are the most promising targets for stabilizing stance judgments.
Meaning-preserving and meaning-inverting rewrites appear to act through the same downstream locus, suggesting a shared mechanism for prompt-sensitivity once a shift occurs.
Combining rewrites amplifies instability beyond individual edits, so evaluations that test only single transformations may understate phrasing sensitivity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A testable extension: if the same syntactic rewrites are applied to non-political factual agreement statements and flips disappear, the constructional effect is specific to opinionated content; if they persist, it is a general readout phenomenon.
The final-position block-output locus suggests stance is read out from a late residual-stream summary rather than from a dedicated early 'stance feature'; probing intermediate representations could confirm whether focus or polarity is encoded separately before the final position.
Benchmark designers could control for cleftability and passivizability when selecting stimulus sentences; otherwise information-structure confounds are built into the data.
Extending to languages with richer morphology (e.g., free word order) could reveal whether the locus shifts when the same constructions are encoded differently.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful, honest empirical study; the behavioral claim about constructional stance shifts is probably right, but control validity and missing error bars on patching keep the localization claim conditional.

read the letter

This is a genuinely useful paper for the NLP robustness crowd. Where prior work used random paraphrases or prompt wording tweaks, this one isolates specific constructional rewrites—passives, it-clefts, wh-clefts, and support verb constructions—and shows they can shift LLM political stance judgments. The dataset extension of ProbVAA is a real contribution, and the variance decomposition cleanly separates articulation sensitivity from sampling noise, which is more than most robustness papers bother to do. The behavioral result is credible: AS values exceed MU across models, the flip rates are sensible, and the larger models are more stable, which matches prior findings. Credit where due: they acknowledge the main limitations in Section 6, including exactly the concern I'd raise first.

That concern is the 'meaning-preserving' rewrite controls. The stress-test note is on target. It-clefts and wh-clefts bring presuppositions and focus effects; the passive example in Table 1 ('Vaccination certificate should be able to be requested...') is degraded at best; and SVC conversions can shift aspect or predicate semantics. The paper concedes in Section 6 that small semantic changes might still occur. This matters because both the behavioral claim and the patching localization inherit whatever artifacts the rewrites carry. I don't think it's disqualifying—the effect sizes are modest, and the meaning-reversing rewrites show larger shifts, which is consistent with the controls being mostly okay—but it needs harder evidence than author manual validation. A human annotation study on truth-condition preservation, or a stricter generation pipeline with round-trip diagnostics, would strengthen the revision.

The activation patching results are suggestive rather than conclusive. The abstract's claim about mid-to-late decoder block outputs at the final prompt position is supported by the mean restoration scores in Figures 3 and 4, but there are no confidence intervals or per-example variance, and the exclusion of low-flip rewrite types per model is post hoc. The intervention is also limited to the final token position, which is honest but narrow. All of this makes the mechanistic conclusion conditional, not broken.

The GitHub link is unverifiable and should be fixed. That's minor.

Who gets value from this? Anyone working on LLM robustness, political stance evaluation, or activation patching. It's a solid empirical building block, not a breakthrough. It deserves a serious referee: the dataset is reusable, the question is well-motivated, and the flaws are addressable in revision. I'd send it to peer review and ask for stronger validation of the rewrite controls and uncertainty quantification on the patching results.

Referee Report

3 major / 5 minor

Summary. The paper studies whether controlled linguistic rewrites—both meaning-preserving (passive, it-cleft, wh-cleft, support verb construction) and meaning-inverting (negation, semantic opposite)—systematically change LLM political stance judgments, and where in the model these changes originate. Using four open-weight models and an extension of the ProbV AA dataset, the authors report flip rates, variance decompositions (PS/AS/MU), and activation patching results. They conclude that stance is sensitive to constructional choice beyond propositional content, and that mid-to-late decoder block outputs at the final prompt position provide the strongest restoration signal.

Significance. If the findings are robust, the paper would make a useful contribution to both LLM robustness evaluation and mechanistic interpretability: it moves beyond random paraphrases to controlled constructional variants, provides a reliability-oriented methodology (flip rates, Wasserstein distances, variance decomposition), and applies activation patching to a meaning-sensitive social-science task. The release of the dataset extension and the explicit prompts are strengths. However, the central behavioral claim—that meaning-preserving rewrites shift stance—rests on the validity of the generated rewrites, and the patching claim is currently reported without uncertainty quantification and is based on a restricted subset of rewrite types for the larger models. These issues are load-bearing for the paper's main conclusions.

major comments (3)

[§3.2, §6, Table 1] The central RQ1 claim—that constructional choice alone shifts stance for 'meaning-preserving' rewrites—depends on the generated rewrites being truth-conditionally equivalent to the base statements. The rewrites are produced by Qwen3-8B few-shot prompting and manually validated only by the authors; Section 6 concedes that 'small semantic changes might still occur for linguistic rewrites.' This is not a formality: the Table 1 passive example ('Vaccination certificate should be able to be requested...') is arguably not equivalent to the original modal/argument structure, and it-clefts/wh-clefts introduce focus presuppositions. If part of the AS/flip-rate signal for the preserving group comes from content drift, the conclusion that stance is sensitive to grammatical realization is undermined. Please report inter-annotator agreement or independent validation, and provide a per-item analysis e
[§4.2, Figures 3/4, Appendix D] The activation patching results are reported as mean normalized restoration scores without confidence intervals, per-example variance, or significance tests. Moreover, the analysis includes only a subset of rewrite types per model: for Gemma-3-12B and Qwen3-14B only negation and opposite; for Qwen3-4B negation, opposite, and it-clefts. The abstract and §5 generalize to 'mid-to-late decoder layers, especially block outputs at the final prompt position' across all rewrites, but the meaning-preserving constructional rewrites are absent from the patching analysis for the two larger models. The claim should be restricted to the analyzed rewrite types, and the patching results need error bars or a per-example distribution to support the localization conclusion.
[§3.6, §4.2, RQ2] RQ2 asks where rewrites 'causally determine' the response shift, and §5 states that the shifts 'can be causally localized to specific internal components.' However, the method establishes only sufficiency of restoring clean behavior by patching activations at the final prompt position, not necessity, and it does not trace the origin of the rewrite-sensitive computation across earlier positions. Section 6 correctly notes that the analysis localizes 'sufficient restoration sites rather than the origin.' This caveat should be carried through the abstract and conclusion; as written, 'causally localized' and 'causally determine' overstate the evidence.

minor comments (5)

[Abstract] Grammar: 'stance instability affect' should be 'stance instability affects.'
[Table 1] The passive example is missing the indefinite article before 'vaccination certificate' and the phrasing 'should be able to be requested' is awkward; this example precisely illustrates the semantic-validity concern raised in the major comments and should be revised or replaced.
[§3.2] The manual validation of the generated rewrites is described in one sentence. Please specify how many annotators were involved, whether disagreements were resolved, and what criteria (e.g., truth-conditionality, naturalness) were used.
[Appendix A.1] The JSON output format in the prompts is malformed in several places (e.g., stray backticks and inconsistent field names like 'variants' vs 'variant'). This makes replication harder; please clean up the prompt listings.
[§4.1] For the flip-rate calculation, pairs with unclear polarity are excluded. Given the low clear-leaning rates for the larger models (40–56%), it would be helpful to report the number of eligible pairs per model and variant, since the flip rates are computed on very different denominators.

Circularity Check

0 steps flagged

No significant circularity; the analysis is empirical and self-contained, with only a minor non-load-bearing self-citation to the base dataset.

full rationale

Walking the derivation chain: (i) The dataset extension is generated by Qwen3-8B prompting and manual validation; the meaning-preserving vs. meaning-inverting labels are input design decisions, not outputs of the analysis. Section 6 concedes that 'small semantic changes might still occur for linguistic rewrites' — this is a stimulus-control validity caveat, not circularity, because no equation or fitted parameter forces the behavioral or patching conclusions from that assumption. (ii) Flip rates, variance decomposition (PS/AS/MU), Wasserstein distance, and the normalized restoration score R = 1 - d_int/d0 are all computed from model output distributions; none is a parameter fit to the quantity it later 'predicts'. (iii) Activation patching is a causal intervention with cached activations; concluding that block outputs at the final prompt position give the strongest restoration is an empirical measurement, and the paper itself explains why this is architecturally expected from the residual stream, so it is not a result smuggled in by definition. The mid-to-late-layer localization is not guaranteed by the method and is a genuine empirical finding. (iv) The only shared-author citation is Ceron et al. (2024), used as the base dataset and prior reliability metric; it is published prior data, not a load-bearing uniqueness theorem or ansatz. No circular step can be exhibited from the paper's equations or citation chain, so the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central empirical claims rest on the validity of the controlled rewrites and on the final-position readout/patching assumptions. The only hand-chosen numerical threshold is the neutrality margin. No new physical or theoretical entities are introduced.

free parameters (1)

Neutrality margin δ = 0.1 on the [-1,1] stance scale
Hand-chosen 'practically negligible region' around neutral defines clear-leaning and therefore which pairs enter flip-rate and patching analyses. No sensitivity analysis is reported.

axioms (4)

domain assumption The six rewrite families preserve or invert truth-conditional meaning as labeled; generated variants do not introduce systematic semantic artifacts.
Section 3.2 and Appendix A.1 generate variants with Qwen3-8B few-shot prompting and manual validation; Section 6 concedes small semantic changes might still occur.
domain assumption The stance distribution is adequately captured by the next-token logits of the seven Likert-digit tokens at the final prompt position.
Section 3.3 restricts the response readout to digit logits z_D(x); no evidence is given that this single-position readout fully represents stance computation.
domain assumption Activation patching at the final prompt position is a valid causal localization method even when base and variant prompts differ in tokenization.
Section 3.5 intervenes only at the final prompt position to avoid alignment assumptions; this may miss rewrite effects encoded at earlier token positions.
standard math The variance decomposition into PS, AS, MU follows Kunievsky and Evans (2026) and is an appropriate additive characterization of response variation.
Appendix B.2 defines PS/AS/MU; the interpretation as additive variance components is assumed without a proof of additivity.

pith-pipeline@v1.3.0-alltime-deepseek · 12479 in / 12098 out tokens · 124463 ms · 2026-08-01T10:42:53.117162+00:00 · methodology

0 comments

read the original abstract

Large language models (LLMs) are known to be sensitive to prompt and input formulations. However, existing studies have focused on lexical realization and largely ignored constructional choice. This paper studies whether linguistic construction can systematically shift LLM decisions and where these shifts can be causally localized inside the model. We use political stance judgment as a meaning-sensitive case study and extend an English political statements dataset, resulting in six controlled linguistic rewrite types that preserve or invert the meaning of a statement. Experiments on four open-weight models show that stance instability affect both meaning-preserving and meaning-inversing rewrites. Because output shifts reveal that rewrites affect stance, but not where in the model, we apply activation patching, where activations from the original statement are substituted into the forward pass for the rewritten statement and measure which components recover the original stance distribution. The results show that mid-to-late decoder layers, especially block outputs at the final prompt position, provide the strongest restoration signal.

Figures

Figures reproduced from arXiv: 2607.20115 by Franziska Weeber, Langchen Huang, Sebastian Pad\'o.

**Figure 2.** Figure 2: Clear-leaning rate and flip rate results. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Activation patching results of Gemma-3-12B-IT (first row) and Gemma-3-4B-IT (second to fourth row). [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Activation patching results of the Qwen3-4B and Qwen3-14B model. The figures in the first row and the [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

23 extracted references · 4 canonical work pages

[1]

Hamidreza Amirzadeh, Afra Alishahi, and Hosein Mohebbi. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.21 How language models prioritize contextual grammatical cues? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 315--336, Miami, Florida, US. Association for Computational Linguistics

work page doi:10.18653/v1/2024.blackboxnlp-1.21 2024
[2]

Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.600 Measuring political bias in large language models: What is said and how it is said . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142--11159, Bangkok, Thailand. Associatio...

doi:10.18653/v1/2024.acl-long.600 2024
[3]

Fabienne Cap, Manju Nirmal, Marion Weller, and Sabine Schulte im Walde. 2015. https://doi.org/10.3115/v1/W15-0903 How to account for idiomatic G erman support verb constructions in statistical machine translation . In Proceedings of the 11th Workshop on Multiword Expressions, pages 19--28, Denver, Colorado. Association for Computational Linguistics

work page doi:10.3115/v1/w15-0903 2015
[4]

Tanise Ceron, Neele Falk, Ana Bari \'c , Dmitry Nikolaev, and Sebastian Pad \'o . 2024. https://doi.org/10.1162/tacl_a_00710 Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLM s . Transactions of the Association for Computational Linguistics, 12:1378--1400

doi:10.1162/tacl_a_00710 2024
[5]

Patrick Haller, Jannis Vamvas, and Lena Ann J \"a ger. 2024. https://openreview.net/pdf?id=7xUtka9ck9 Yes, no, maybe? revisiting language models' response stability under paraphrasing for the assessment of political leaning . In First Conference on Language Modeling

2024
[6]

Myeongjun Jang and Thomas Lukasiewicz. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.991 Consistency analysis of C hat GPT . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15970--15985, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.991 2023
[7]

Sullam Jeoung, Yubin Ge, Haohan Wang, and Jana Diesner. 2025. https://openreview.net/forum?id=7LGmXXZXtP Examining alignment of large language models through representative heuristics: the case of political stereotypes . In The Thirteenth International Conference on Learning Representations

2025
[8]

Juho Kim, Christopher Malon, and Asim Kadav. 2018. https://doi.org/10.18653/v1/W18-5512 Teaching syntax by adversarial distraction . In Proceedings of the First Workshop on Fact Extraction and VER ification ( FEVER ) , pages 79--84, Brussels, Belgium. Association for Computational Linguistics

work page doi:10.18653/v1/w18-5512 2018
[9]

Nadav Kunievsky and James A. Evans. 2026. https://openreview.net/forum?id=qhFEKDeBsO Measuring intent comprehension in LLM s . In The Forty-third International Conference on Machine Learning

2026
[10]

Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.12 Linguistically-informed transformations ( LIT ): A method for automatically generating contrast sets . In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pag...

doi:10.18653/v1/2020.blackboxnlp-1.12 2020
[11]

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf Locating and editing factual associations in GPT . Advances in neural information processing systems, 35:17359--17372

2022
[12]

Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. https://doi.org/10.18653/v1/S16-1001 S em E val-2016 task 4: Sentiment analysis in T witter . In Proceedings of the 10th International Workshop on Semantic Evaluation ( S em E val-2016) , pages 1--18, San Diego, California. Association for Computational Linguistics

doi:10.18653/v1/s16-1001 2016
[13]

Ansh Poonia and Maeghal Jain. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.1335 Dissecting persona-driven reasoning in language models via activation patching . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24553--24566, Suzhou, China. Association for Computational Linguistics

doi:10.18653/v1/2025.findings-emnlp.1335 2025
[14]

Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. https://doi.org/10.18653/v1/2024.acl-long.816 Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models . In Proceedings of the 62nd Annual Meeting of the Association for ...

doi:10.18653/v1/2024.acl-long.816 2024
[15]

Jens Rupprecht, Georg Ahnert, and Markus Strohmaier. 2026. https://doi.org/10.18653/v1/2026.nlpcss-1.1 Prompt perturbations reveal human-like biases in large language model survey responses . In Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science, pages 1--21, San Diego. Association for Computational Linguistics

work page doi:10.18653/v1/2026.nlpcss-1.1 2026
[16]

Tak, Amin Banayeeanzade, Anahita Bolourani, Mina Kian, Robin Jia, and Jonathan Gratch

Ala N. Tak, Amin Banayeeanzade, Anahita Bolourani, Mina Kian, Robin Jia, and Jonathan Gratch. 2025. https://doi.org/10.18653/v1/2025.findings-acl.679 Mechanistic interpretability of emotion inference in large language models . In Findings of the Association for Computational Linguistics: ACL 2025, pages 13090--13120, Vienna, Austria. Association for Compu...

doi:10.18653/v1/2025.findings-acl.679 2025
[17]

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. https://doi.org/10.48550/arXiv.2...

Pith/arXiv arXiv doi:10.48550/arxiv.2503.19786 2025
[18]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . Advances in neural information processing systems, 30

2017
[19]

Yuqing Wang and Yun Zhao. 2024. https://arxiv.org/pdf/2406.11020 Rupbench: Benchmarking reasoning under perturbations for robustness evaluation in large language models . arXiv preprint arXiv:2406.11020

Pith/arXiv arXiv 2024
[20]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/pdf/2505.09388? Qwen3 technical report . arXiv preprint arXiv:2505.09388

Pith/arXiv arXiv 2025
[21]

Fred Zhang and Neel Nanda. 2024. https://openreview.net/forum?id=Hf17y6u9BC Towards best practices of activation patching in language models: Metrics and methods . In The Twelfth International Conference on Learning Representations

2024
[22]

Yanlu Zhong, Simon Todd, Nicole Xu, and Laurel Brehm. 2025. https://www.sciencedirect.com/science/article/pii/S2772766125000953 Evaluating llms as proxies for humans in psycholinguistic ratings: A comparison of statistical knowledge . Research Methods in Applied Linguistics, 4(3):100274

2025
[23]

Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.654 On the calibration of large language models and alignment . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9778--9795, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.findings-emnlp.654 2023

[1] [1]

Hamidreza Amirzadeh, Afra Alishahi, and Hosein Mohebbi. 2024. https://doi.org/10.18653/v1/2024.blackboxnlp-1.21 How language models prioritize contextual grammatical cues? In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 315--336, Miami, Florida, US. Association for Computational Linguistics

work page doi:10.18653/v1/2024.blackboxnlp-1.21 2024

[2] [2]

Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. 2024. https://doi.org/10.18653/v1/2024.acl-long.600 Measuring political bias in large language models: What is said and how it is said . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11142--11159, Bangkok, Thailand. Associatio...

doi:10.18653/v1/2024.acl-long.600 2024

[3] [3]

Fabienne Cap, Manju Nirmal, Marion Weller, and Sabine Schulte im Walde. 2015. https://doi.org/10.3115/v1/W15-0903 How to account for idiomatic G erman support verb constructions in statistical machine translation . In Proceedings of the 11th Workshop on Multiword Expressions, pages 19--28, Denver, Colorado. Association for Computational Linguistics

work page doi:10.3115/v1/w15-0903 2015

[4] [4]

Tanise Ceron, Neele Falk, Ana Bari \'c , Dmitry Nikolaev, and Sebastian Pad \'o . 2024. https://doi.org/10.1162/tacl_a_00710 Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLM s . Transactions of the Association for Computational Linguistics, 12:1378--1400

doi:10.1162/tacl_a_00710 2024

[5] [5]

Patrick Haller, Jannis Vamvas, and Lena Ann J \"a ger. 2024. https://openreview.net/pdf?id=7xUtka9ck9 Yes, no, maybe? revisiting language models' response stability under paraphrasing for the assessment of political leaning . In First Conference on Language Modeling

2024

[6] [6]

Myeongjun Jang and Thomas Lukasiewicz. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.991 Consistency analysis of C hat GPT . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15970--15985, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.emnlp-main.991 2023

[7] [7]

Sullam Jeoung, Yubin Ge, Haohan Wang, and Jana Diesner. 2025. https://openreview.net/forum?id=7LGmXXZXtP Examining alignment of large language models through representative heuristics: the case of political stereotypes . In The Thirteenth International Conference on Learning Representations

2025

[8] [8]

Juho Kim, Christopher Malon, and Asim Kadav. 2018. https://doi.org/10.18653/v1/W18-5512 Teaching syntax by adversarial distraction . In Proceedings of the First Workshop on Fact Extraction and VER ification ( FEVER ) , pages 79--84, Brussels, Belgium. Association for Computational Linguistics

work page doi:10.18653/v1/w18-5512 2018

[9] [9]

Nadav Kunievsky and James A. Evans. 2026. https://openreview.net/forum?id=qhFEKDeBsO Measuring intent comprehension in LLM s . In The Forty-third International Conference on Machine Learning

2026

[10] [10]

Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. https://doi.org/10.18653/v1/2020.blackboxnlp-1.12 Linguistically-informed transformations ( LIT ): A method for automatically generating contrast sets . In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pag...

doi:10.18653/v1/2020.blackboxnlp-1.12 2020

[11] [11]

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/6f1d43d5a82a37e89b0665b33bf3a182-Paper-Conference.pdf Locating and editing factual associations in GPT . Advances in neural information processing systems, 35:17359--17372

2022

[12] [12]

Preslav Nakov, Alan Ritter, Sara Rosenthal, Fabrizio Sebastiani, and Veselin Stoyanov. 2016. https://doi.org/10.18653/v1/S16-1001 S em E val-2016 task 4: Sentiment analysis in T witter . In Proceedings of the 10th International Workshop on Semantic Evaluation ( S em E val-2016) , pages 1--18, San Diego, California. Association for Computational Linguistics

doi:10.18653/v1/s16-1001 2016

[13] [13]

Ansh Poonia and Maeghal Jain. 2025. https://doi.org/10.18653/v1/2025.findings-emnlp.1335 Dissecting persona-driven reasoning in language models via activation patching . In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 24553--24566, Suzhou, China. Association for Computational Linguistics

doi:10.18653/v1/2025.findings-emnlp.1335 2025

[14] [14]

Paul R \"o ttger, Valentin Hofmann, Valentina Pyatkin, Musashi Hinck, Hannah Kirk, Hinrich Schuetze, and Dirk Hovy. 2024. https://doi.org/10.18653/v1/2024.acl-long.816 Political compass or spinning arrow? towards more meaningful evaluations for values and opinions in large language models . In Proceedings of the 62nd Annual Meeting of the Association for ...

doi:10.18653/v1/2024.acl-long.816 2024

[15] [15]

Jens Rupprecht, Georg Ahnert, and Markus Strohmaier. 2026. https://doi.org/10.18653/v1/2026.nlpcss-1.1 Prompt perturbations reveal human-like biases in large language model survey responses . In Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science, pages 1--21, San Diego. Association for Computational Linguistics

work page doi:10.18653/v1/2026.nlpcss-1.1 2026

[16] [16]

Tak, Amin Banayeeanzade, Anahita Bolourani, Mina Kian, Robin Jia, and Jonathan Gratch

Ala N. Tak, Amin Banayeeanzade, Anahita Bolourani, Mina Kian, Robin Jia, and Jonathan Gratch. 2025. https://doi.org/10.18653/v1/2025.findings-acl.679 Mechanistic interpretability of emotion inference in large language models . In Findings of the Association for Computational Linguistics: ACL 2025, pages 13090--13120, Vienna, Austria. Association for Compu...

doi:10.18653/v1/2025.findings-acl.679 2025

[17] [17]

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and 197 others. 2025. https://doi.org/10.48550/arXiv.2...

Pith/arXiv arXiv doi:10.48550/arxiv.2503.19786 2025

[18] [18]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . Advances in neural information processing systems, 30

2017

[19] [19]

Yuqing Wang and Yun Zhao. 2024. https://arxiv.org/pdf/2406.11020 Rupbench: Benchmarking reasoning under perturbations for robustness evaluation in large language models . arXiv preprint arXiv:2406.11020

Pith/arXiv arXiv 2024

[20] [20]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://arxiv.org/pdf/2505.09388? Qwen3 technical report . arXiv preprint arXiv:2505.09388

Pith/arXiv arXiv 2025

[21] [21]

Fred Zhang and Neel Nanda. 2024. https://openreview.net/forum?id=Hf17y6u9BC Towards best practices of activation patching in language models: Metrics and methods . In The Twelfth International Conference on Learning Representations

2024

[22] [22]

Yanlu Zhong, Simon Todd, Nicole Xu, and Laurel Brehm. 2025. https://www.sciencedirect.com/science/article/pii/S2772766125000953 Evaluating llms as proxies for humans in psycholinguistic ratings: A comparison of statistical knowledge . Research Methods in Applied Linguistics, 4(3):100274

2025

[23] [23]

Chiwei Zhu, Benfeng Xu, Quan Wang, Yongdong Zhang, and Zhendong Mao. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.654 On the calibration of large language models and alignment . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9778--9795, Singapore. Association for Computational Linguistics

doi:10.18653/v1/2023.findings-emnlp.654 2023