Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper claims that in several LLMs, moral value contaminates grammatical and economic judgments relative to humans, and that ablating a morality direction in the residual stream repairs the contamination.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Some LLMs conflate moral value with grammatical and economic value, and ablating a morality direction in activations partially repairs grammar and economic judgments.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection The behavioral core—grammar and economic ratings tracking moral content in several LLMs—looks real; the causal 'morality vector' ablation is less clean than the abstract claims. the 4 major comments →

arxiv 2602.19101 v2 pith:IGQFZSHH submitted 2026-02-22 cs.CL cs.AI

Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models

Seong Hah Cho , Junyi Li , Anna Leshinskaya This is my paper
classification cs.CL cs.AI
keywords value entanglementvalue alignmentmoral valuegrammaticalityeconomic valuerepresentation engineeringdirectional ablationresidual stream activations
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models distinguish different kinds of good — moral, grammatical, and economic — the way humans do. It finds that in several models (GPT-3.5, Qwen2.5-7B-Instruct, Gemma-2-9B-Instruct, Mistral-Small-24B-Instruct), grammaticality and economic-value ratings strongly track moral content: a well-formed sentence describing a moral wrong gets rated as ungrammatical, and an expensive watch mentioned in a harmful scenario is rated cheap. Humans show essentially no such correlation (r=0.05). The paper localizes the conflation to overlapping attribute-vector directions in the models' internal activations, and shows that ablating the morality direction during inference improves grammar and economic ratings, while newer closed-source models (GPT-4o mini, Gemini 2.0, Claude 3.7) do not behave as if entangled. The authors argue this 'value entanglement' matters for alignment because a model that cannot separate kinds of good will produce distorted evaluations across domains.

Core claim

The central discovery is that moral value and other value dimensions are entangled in the internal representations of some language models: the direction in activation space that codes for moral goodness overlaps with the directions for grammatical goodness and economic value, so that projecting a sentence onto a grammar vector partially recovers its moral content. Behaviorally, this means model ratings of grammaticality correlate almost as strongly with human moral ratings (r=0.56 for GPT-3.5) as with human grammaticality ratings (r=0.70), while in humans the same correlation is r=0.05. The entanglement is causal in the specific sense that directional double-ablation of the morality vector

What carries the argument

The carrying object is a set of attribute vectors — for morality, grammaticality, and economic value — estimated by the difference-of-means method: subtracting averaged residual-stream activations (or embeddings) of positive examples from negative examples at the last token position (Eq. 1). Test sentences are located along each axis by inner-product projection (Eq. 3). The causal claim rests on directional ablation (Eq. 4), which zeroes or flips variance along the morality direction during inference, and shows that grammar and price ratings become more human-like. The same recipe is applied to closed models via embedding similarity.

Load-bearing premise

The load-bearing premise is that the contrast pairs used to build the morality vector isolate moral value rather than a broader dimension such as danger, negativity, or vividness; if that premise fails, the ablation results show only that some negativity-like signal interferes with grammatical and economic judgment, not moral value specifically.

What would settle it

Re-run the grammar-rating task with non-moral negative sentences (e.g., 'I stepped on a slug with my bare foot' with 0 to 4+ grammatical errors). If grammar ratings still drop with moral-content-like magnitude, the interference is general negativity rather than moral value. Alternatively, construct morality contrast pairs that hold the action constant and vary only the stated intention (e.g., 'I gave the patient the medicine to save her' vs. 'I gave the patient the medicine to silence her'), rebuild the morality direction from those, and check whether ablating it still repairs grammar and pric

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Entanglement is not an artifact of instruction tuning alone: pre-trained-only base variants of the three open-weight models already show significant cross-domain correlations between moral and grammar/economic projections.
  • Selective ablation of the morality direction improves grammaticality ratings (closer to human norms) and economic ratings (closer to ground-truth prices) in Qwen2.5-7B-Instruct, Gemma-2-9B-Instruct, and Mistral-Small-24B-Instruct.
  • Inference-time interventions on the morality vector also lower the correlation with independent human moral norms, confirming that the ablated direction carries moral information.
  • Behavioral entanglement is absent in GPT-4o mini, Gemini 2.0, and Claude 3.7 Sonnet, yet their embedding models still show conflation, suggesting representations may remain entangled even when behavior is clean.
  • The authors define value entanglement as an evaluable alignment failure: value functions mapping sentences to scalar goodness are warped relative to human norms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the morality contrast pairs (such as smuggling clean water versus detonating a nuclear device) differ in danger, legality, and vividness, the ablated direction may capture general negativity rather than morality specifically; testing with matched non-moral negative sentences would separate these accounts.
  • The projection-plus-ablation template could be applied to other value kinds — aesthetic, epistemic, hedonic — to map the full geometry of 'good' in language models.
  • If closed embedding models harbor the conflation while closed chat models behave cleanly, alignment procedures may be masking, not erasing, entanglement; adversarial inputs or fine-tuning might re-expose it.
  • An entanglement metric of this kind could serve as an evaluation signal during training: check whether grammar and price judgments move with moral content, and use the cross-correlation as a loss term.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper tests whether LLMs distinguish three kinds of value — moral, grammatical, and economic — by creating stimulus sets that vary these dimensions orthogonally (MoralGrammar68 and MoralEconomic68), validating them with human ratings, and probing models behaviorally, through embeddings, and via residual-stream activations. Behavioral ratings show that in GPT-3.5, Qwen2.5-7B-Instruct, Gemma-2-9B-Instruct, and Mistral-Small-24B-Instruct, grammaticality and economic-value judgments are overly influenced by moral content relative to human norms, while newer closed-weight models do not show this effect. Residual-stream projections in the open-weight models show correlations between moral and grammatical/economic directions, and directional ablation of a 'morality' vector is reported to improve grammaticality and economic-value rating correlations. The paper also compares base and instruction-tuned variants to assess training contributions.

Significance. If the central claim holds, the paper makes a valuable contribution: it provides a quantitative, multi-model behavioral demonstration that some LLMs conflate distinct kinds of value, which is directly relevant to alignment. The behavioral evidence (Table 2; Section 3.1) is robust, uses human norms as ground truth, and is independent of the internal-vector construction. The open-weight analyses and ablation experiments are a useful first step toward a mechanistic account, and the appendices are transparent about layer-wise statistics. However, the causal 'selective ablation' and 'repair' claims rest on the construct validity of the morality direction, which is not yet established; the embedding analyses also face a potential circularity concern. These issues are addressable but require additional controls and statistical clarifications.

major comments (4)
  1. [Section 2.3, Eq. (1); Appendix B.1] The morality direction d(l) is computed from 48 positive vs. 48 negative moral scenarios (e.g., smuggling clean water vs. detonating a nuclear device). These contrast pairs differ not only in moral valence but also in physical danger, illegality, vividness, and atypicality. The claim that ablating this direction 'selectively' removes moral information (abstract; Section 3.4) is therefore not yet supported; the results could reflect removal of a broader harmfulness/negativity signal. The Animal Size control (Section 2.4) checks an unrelated attribute and cannot rule out these confounds. The authors should either match contrast pairs on non-moral dimensions or demonstrate that d(l) is orthogonal to directions for danger/legality/vividness, before claiming moral-specificity.
  2. [Section 3.2] The grammaticality embedding vector is constructed from adjectives including 'well written' vs. 'poorly written' alongside purely syntactic contrasts. These valenced adjectives may share a common good/bad axis with the morality vector, which uses 'morally virtuous' vs. 'morally wrong'. The reported high correlation between the moral and grammaticality vectors (r=.58) and the strong correlation between grammaticality projections and human morality ratings (r=.68) may therefore be an artifact of the adjective contrasts rather than evidence of entangled representations. The authors should re-run the embedding analysis with strictly syntactic contrasts (e.g., 'syntactically well-formed' vs. 'syntactically ill-formed') or with non-valenced control adjectives.
  3. [Sections 3.1 and 3.3] The ANOVA reporting is internally inconsistent. For binned data with 3 morality levels and 4 grammar/economic levels, a two-way ANOVA would yield F(2,·) and F(3,·) degrees of freedom, but the text reports F(1,64) throughout (e.g., GPT-3.5 grammaticality: F(1,64)=109.51 for the morality effect). This suggests a linear-contrast or regression model, not the described ANOVA; please clarify and correct the dfs. Additionally, the residual-stream ANOVA is performed only at the peak-correlation layer selected post hoc (Qwen l21, Gemma l10, Mistral l40; Section 3.3) with no multiple-comparison correction across layers. The appendix shows a broad effect, but the main-text p-values should be corrected for layer selection or the analysis pre-registered.
  4. [Section 3.5] The pre/post-training comparison reports Δr=1.07 (Mistral-Small, MoralGrammar68 l21) and Δr=1.31 (Mistral-Small, MoralEconomic68 l18) as 'differences of correlations.' Pearson correlations are bounded in [-1,1], so a difference of two correlations cannot exceed 2, and values above 1 are impossible under the standard interpretation. If these are differences of Fisher z-transformed correlations, the text must say so explicitly. The direction of the difference (which variant has the higher correlation) should also be stated for Gemma-2 and Mistral-Small.
minor comments (5)
  1. [Appendix C.2] The header of the MoralEconomic68 projection statistics table says 'Grammaticality Projections' in place of 'Economic Projections'; this appears to be a copy-paste error from C.1 and should be corrected.
  2. [Section 2.4] Typo: 'MoralGrammer68' should be 'MoralGrammar68'.
  3. [Section 3.1] Incomplete sentence: 'Similar effects held in and Gemma-2 9B. Mistral-Small 24B.' — grammar needs fixing.
  4. [Equation (3)] The notation x(l)−1(t) is not defined; specify that it is the residual-stream activation at the last token position of input t at layer l.
  5. [Figure 3] Asterisks indicate significance but the figure caption does not state the significance threshold or whether correction was applied across layers; please clarify in the caption.

Circularity Check

1 steps flagged

Embedding evidence of grammar–morality entanglement is partly by construction; behavioral and ablation evidence remain independent.

specific steps
  1. self definitional [Section 2.2 (Embedding vector construction) and Section 3.2 (Embedding results)]
    "For grammaticality, we contrasted the adjectives "syntactically accurate", "grammatical", "well written", "linguistically correct", "syntactically well-formed" minus "syntactically inaccurate", "ungrammatical", "poorly written", "linguistically incorrect", "syntactically ill-formed". ... In the GPT embedding model, vectors for morality and grammaticality were themselves highly correlated at r=.58."

    The grammaticality vector is not purely syntactic: "well written" and "poorly written" are general evaluative phrases carrying positive/negative valence. The moral vector is likewise a contrast of positive vs. negative moral adjectives. Both directions therefore point along a shared good/bad semantic polarity, so the high vector correlation (r=.58) and the cross-correlation of item projections (r=.80; grammaticality projections vs. human morality r=.68) are inflated by the way the axes were defined. The "grammaticality" measure already contains a general goodness component, making part of the reported grammar--morality entanglement an artifact of the operationalization rather than an independent model property. The residual-stream analyses (minimal pairs like "I go" vs. "I goes") and the b

full rationale

The paper's central behavioral finding—that GPT-3.5/Qwen2.5/Gemma grammaticality ratings correlate with human moral ratings almost as strongly as with human grammaticality ratings—is derived from independent human norms, orthogonal stimulus construction, and model prompts; it is not fitted to the outcome. The residual-stream projection and ablation analyses use contrast pairs distinct from the test stimuli and are validated against external moral vignettes (Dillion et al.) and an Animal Size control, so they do not reduce to their inputs. The one partially circular element is the embedding analysis in Section 3.2: the grammaticality direction is built from adjectives including "well written" vs. "poorly written," which are general evaluative terms rather than pure syntax descriptors. Because the moral direction is also a positive-vs-negative contrast, the reported high correlation between the two vectors and between grammaticality projections and human morality is partly guaranteed by the shared valence polarity of the defining word sets. This is a construct-validity confound and a partial self-definitional circularity in a supporting analysis, not in the main behavioral or causal chain. A separate concern—that the morality direction in Eq. (1) is derived from contrast pairs differing in danger, illegality, and atypicality (Appendix B.1)—is a correctness/validity risk for the 'moral-specific' interpretation of the ablation, but it is not a derivation-circularity step because the direction is not fitted to the grammatical/economic outcomes it is used to predict.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on the validity of difference-of-means vectors as attribute isolators, the human normative benchmark, and the linear representational assumption; no new entities are postulated.

free parameters (3)
  • ablation strength alpha = 2 (double ablation)
    Chosen by hand in Section 2.4 because double ablation 'produced more consistent and selective disruptions'; affects the ablation recovery results.
  • analysis layer selection = peak layer per model (e.g., l21 for Qwen2.5 7B)
    Layer with maximal cross-domain correlation selected for ANOVA; no correction for multiple layers (Section 3.3).
  • morality binning = 3 levels
    Binning of morality and grammar into 3/4 levels for ANOVA; arbitrary discretization.
axioms (4)
  • domain assumption Difference-of-means contrast sentences differ only along the target attribute
    Section 2.3 assumes the 48 contrast pairs per domain isolate the attribute; moral contrast pairs also differ in danger, legality, and vividness.
  • domain assumption Human Likert ratings from 67 Prolific participants are an appropriate normative benchmark for value separation
    Human r=.05 used as the null norm; sample size and cultural scope may not generalize (Section 2.1).
  • domain assumption Value attributes are representable as linear directions in activation space (difference-of-means projection)
    Follows prior work (Grand et al., Arditi et al.), but is an unproven assumption for this specific claim.
  • domain assumption The two stimulus sets are orthogonally designed and validated by human ratings
    Human ratings confirm independence of moral and grammar/moral and economic levels (Section 2.1).

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models." pith.science (2026). https://pith.science/paper/IGQFZSHH

@misc{pith2026260219101,
  author       = {Pith},
  title        = {Pith review of: Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGQFZSHH}},
  note         = {Machine review of arXiv:2602.19101}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Value alignment of Large Language Models (LLMs) requires us to empirically measure these models' actual, acquired representation of value. Among the characteristics of value representation in humans is that they distinguish among value of different kinds. We investigate whether LLMs likewise distinguish three different kinds of good: moral, grammatical, and economic. By probing model behavior, embeddings, and residual stream activations, we report pervasive cases of value entanglement: a conflation between these distinct representations of value. Specifically, both grammatical and economic valuation was found to be overly influenced by moral value, relative to human norms. This conflation was repaired by selective ablation of the activation vectors associated with morality.

Figures

Figures reproduced from arXiv: 2602.19101 by Anna Leshinskaya, Junyi Li, Seong Hah Cho.

Figure 1
Figure 1. Figure 1: Model ratings sentences from MoralGrammar68 (top) and MoralEconomic68 (bottom) for closed (GPT-3.5, GPT-4o mini) and open-source (Qwen2.5 7B, Gemma-2 9B, Mistral-Small 24B) models. Center colors indicate morally good (blue), neutral (white), and morally bad (red) scenarios. Edge colors indicate groups of stimuli varying across grammar or economic scales for a single moral scenario. Shapes and their number … view at source ↗
Figure 2
Figure 2. Figure 2: Residual stream activations projections onto the gram￾mar attribute vector from MoralGrammar68 (left) and onto the economic attribute vector from MoralEconomic68 (right) stimulus sets, as a function of 3 morality levels, in GPT-3.5, Qwen2.5 7B, Gemma-2 9B, and Mistral-Small 24B. Error bars show mean (cen￾ter line) ± SEM. found in Gemma-2 9B (l 10: F(2, 56) = 14.02, p < .001) and Mistral-Small 24B (l 40: F(… view at source ↗
Figure 3
Figure 3. Figure 3: Correlation modulations between human and model ratings and object price and model ratings during double ablation intervention using a morality vector (left), grammar vector (middle), and economic vector (right). Asterisks indicate layers where the correlation changes significantly compared to baseline and control questions (Animal Size). 3.5. Influence of Pre-training and Post-training on Entanglement To … view at source ↗
Figure 4
Figure 4. Figure 4: Comparisons of Qwen2.5 7B residual stream activation projections between the pre-trained only and instruction-tuned models on MoralGrammar68 (left) and MoralEconomic68 (right). Presence of markers (square; circle) indicate that the cross-domain correlation (e.g. morality versus grammar projection values) for the corresponding layer is statistically different from 0. Gray shading indicates that the cross-do… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal

    cs.LG 2026-06 unverdicted novelty 5.0

    A new pipeline uses interpretability to characterize concepts in preference data and shape rewards via feature or data interventions during LM post-training.

Reference graph

Works this paper leans on

13 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [5]

    Gemma Team, Riviere, M., Pathak, S., Sessa, P

    URLhttp://arxiv.org/abs/2404.16244. Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thak...

    Pith/arXiv arXiv

  2. [6]

    Leshinskaya, A

    URL http: //arxiv.org/abs/2109.13916. Leshinskaya, A. and Chakroff, A. Value as semantics: rep- resentations of human moral and hedonic value in large language models.AI meets moral philosophy and moral psychology workshop at NeurIPS (, 37,

    Pith/arXiv arXiv

  3. [7]

    URLarXiv:2502.08640

    doi: arXiv:2502.08640. URLarXiv:2502.08640. Mistral AI. Mistral small

    Pith/arXiv arXiv

  4. [8]

    org/abs/2312.06681

    URL http://arxiv. org/abs/2312.06681. Qwen, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren,...

    Pith/arXiv arXiv

  5. [9]

    2412.15115

    doi: 10.48550/arXiv. 2412.15115. URL http://arxiv.org/abs/2412. 15115. arXiv:2412.15115 [cs]. 9 Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models Soligo, A., Turner, E., Rajamanoharan, S., and Nanda, N. Convergent linear representations of emergent misalign- ment, June

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115

  6. [10]

    arXiv:2506.11618 [cs]

    URL http://arxiv.org/abs/ 2506.11618. arXiv:2506.11618 [cs]. Turner, E., Soligo, A., Taylor, M., Rajamanoharan, S., and Nanda, N. Model organisms for emergent misalignment, June

    Pith/arXiv arXiv

  7. [11]

    URL http://arxiv.org/abs/2506. 11613. arXiv:2506.11613 [cs]. Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., Goel, S., Li, N., Byun, M. J., Wang, Z., Mallen, A., Basart, S., Koyejo, S., Song, D., Fredrikson, M., Kolter, J. Z., and Hendrycks, D. Representation engineering: a top- down approach ...

    Pith/arXiv arXiv

  8. [12]

    URL http://arxiv.org/abs/ 2310.01405

    doi: 10.48550/ arXiv.2310.01405. URL http://arxiv.org/abs/ 2310.01405. 10 Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models A. Task Instructions and Model Prompts A.1. Human instructions Morality instructions:In this survey, you will be asked to rate the morality of specific actions. Your ratings will be on a s...

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.01405

  9. [13]

    OLEDinstead of going to the optional work event. Neutral $$$$ I chose to watch TV on myLG 65

    You are an expert judge of the size of animals. Please rate animals on a scale from 0 to 100, where 0 means small/little/tiny and 100 means large/big/huge. Return only a single score between 0 and 100 without any additional text. [Animal]: [Rating]: 12 Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Models B. Stimul...

    1996

  10. [2022]

    Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O

    URL http://arxiv.org/abs/2212.08073. Betley, J., Tan, D., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., and Evans, O. Emergent misalign- ment: narrow finetuning can produce broadly misaligned llms,

    Pith/arXiv arXiv

  11. [2023]

    doi: 10.1016/j.tics.2023

    ISSN 13646613. doi: 10.1016/j.tics.2023. 04.008. URL https://linkinghub.elsevier. com/retrieve/pii/S1364661323000980. Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V ., Iqbal, H., Tomašev, N., Ktena, I., Kenton, Z., Rodriguez, M., El-Sayed, S., Brown, S., Akbulut, C., Trask, A., Hughes, E., Bergman, A. S., Shelby, R., Mar- chal, N., Gri...

    work page doi:10.1016/j.tics.2023 2023

  12. [2024]

    Arturi, D

    URL http: //arxiv.org/abs/2406.11717. Arturi, D. A. R., Zhang, E., Ansah, A., Zhu, K., Panda, A., and Balwani, A. Shared parameter subspaces and cross-task linearity in emergently misaligned behav- ior,

    Pith/arXiv arXiv

  13. [2025]

    URL http://arxiv.org/abs/2511. 02022. arXiv:2511.02022 [cs]. Bai, Y ., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McK- innon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Luk...

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.