REVIEW 5 major objections 6 minor 2 references
Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Large language models often make choices in concrete scenarios that contradict the principles they endorsed in abstract prompts, and a single contextual cue such as the decision-maker's gender can trigger the flip.
desk verdict Useful stated/revealed preference lens, but the headline claim rests on an unvalidated mapping between abstract principles and contextual choices. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing framework is the stated-versus-revealed preference pair, borrowed from economics and applied to prompts. A base prompt, together with ten paraphrases, asks the model to endorse one of two competing principles — say, contribution-based fairness versus outcome-based equality — and the majority selection fixes the stated, or prior, distribution over principles. A set of contextualized forced binary choices, each deliberately manipulating a single psychological factor (gender of the decision-maker, probability of an outcome, peer behavior, role perspective, gain/loss framing), then yields the posterior, context-conditioned distribution. The KL divergence between these two distributions, alongside the absolute deviation of the dominant principle's probability, is the operational measure of how far revealed behavior sits from stated principle. The appendix's assertion that each contextual choice 'represents a semantic mapping to one of the principles previously tested' is the bridge that makes the comparison meaningful.
What would settle it
Ask independent annotators, without seeing the base prompt, to label each contextualized option with the principle it embodies. If a substantial share of options cannot be mapped to one of the two named principles — for instance, the base prompt asks whether people prioritize attractiveness over money while the scenario asks whom a specific person would date — then the reported deviation is partly an artifact of comparing different questions, and the headline numbers should shrink once the base and contextual prompts are aligned in question type (descriptive belief versus personal decision).
Extended reading notes
Core claim
On the paper's own terms, the central claim is that today's LLMs exhibit a systematic, measurable divergence between stated and revealed preferences: the guiding principle a model endorses in a general prompt is frequently not the principle governing its choice in a contextualized scenario. The authors define a preference deviation as a change in the dominant principle — the arg-max over principles shifts once the response is conditioned on a concrete context — and quantify it with the absolute difference in adoption probabilities and with KL divergence between the prior and context-conditioned distributions. Their experiments show that small contextual manipulations, such as changing the decision-maker's gender, adding peer influence, switching a gain frame to a loss frame, or altering the stakeholder's role, often pivot the preferred choice. In one illustrative case, GPT justifies its dating choice by appealing to compatibility when the actual driving factor, absent from the model's own explanation, is the gender of the decision-maker. On aggregate, GPT shows the deeper internal shifts (KL divergence 0.697 overall) while Gemini shows weaker internal but still sizable surface variability (0.424), and the paper's claim is that this instability appears across every preference category and both models.
Load-bearing premise
The entire measurement depends on the assumption that each contextualized choice option maps cleanly onto one of the two principles tested in the abstract base prompt, so that choosing a different option counts as reversing the same preference; this mapping is asserted in the appendix but is never validated by human annotation, by the models' own explanations, or by an external benchmark.
Editorial extensions
If this is right
- Alignment evaluations that rely on a model's answers to abstract ethical or normative questions cannot be trusted to predict its behavior in concrete applications; the two can diverge by an average absolute probability of about 0.37 in this test.
- Deployment in high-stakes settings — legal, medical, and financial — needs context-level auditing, because a single contextual cue can silently switch the operative principle without the model's explanation revealing it.
- Models differ in how they deviate: GPT's larger KL divergence indicates deeper shifts in the underlying principle distribution, whereas Gemini's deviations are more surface-level, so a single summary score can hide which kind of instability a model has.
- The 84% neutrality rate of Claude on abstract prompts means that some models avoid committing to stated principles altogether, making stated-preference auditing impossible for them even though they still behave in context.
- The dataset doubles as a benchmark: others can run the same 23 base prompts and contextual variants to check whether future models close the stated-revealed gap.
Reading between the lines
- A testable extension: the same protocol could be run with human annotators labeling each contextualized option by the principle it instantiates; if the labels disagree with the authors' mapping, part of the measured deviation is an artifact of prompt design rather than a property of the models.
- If the mapping does hold, the results connect to the broader finding that LLM behavior is highly sensitive to surface form; one implication is that preference 'alignment' may need to be defined at the level of the choice distribution over scenarios, not over principles.
- The gender-driven flip that the model justifies as 'compatibility' suggests that explanation generation can rationalize a decision after the fact; a practical follow-up would ask the model to predict its own choice with and without each contextual cue to see whether its self-insight improves.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a formal definition of the deviation between an LLM's stated preferences (responses to abstract, general prompts) and its revealed preferences (choices in contextualized binary-choice scenarios), and it introduces two metrics—absolute probability deviation and KL divergence—to quantify this deviation. The authors construct 23 base prompts with 10 paraphrases each and 4–24 contextual variants per category, evaluate GPT and Gemini through commercial APIs, and report that overall absolute deviations are 0.371 and 0.355 respectively. Claude is excluded because of a high neutrality rate. The paper claims that minor prompt-format changes often pivot the preferred choice across preference categories and models.
Significance. If the measurement were valid, the proposed framework and prompt set could serve as a useful benchmark for probing LLM alignment consistency. The paper contributes explicit formal definitions (Eqs. 1–7), a structured prompt design grounded in behavioral-economics paradigms, and an openly described dataset. However, the current evidence does not establish the headline claim: the semantic mapping between base and contextual choices is asserted but not validated, all probabilities are frequencies over small samples without uncertainty quantification, and the model coverage claim is contradicted by the post-hoc exclusion of Claude. The conceptual distinction between stated and revealed preferences is interesting and worth pursuing, but the paper needs substantial additional validation before its central claim can be accepted.
major comments (5)
- [Appendix A and Figure 1] The measurement assumes that each contextualized option maps unambiguously to one of the two principles tested in the base prompt, but this mapping is asserted, not validated. For example, the base prompt 'Do you think people prioritize physical attractiveness over financial success...' asks for a descriptive belief about people, while the contextual prompt 'Who do you like?' asks for a first-person decision; similarly, 'should language models avoid gender stereotypes...' is a normative question, while 'The nurse walked into the room. What did _ say?' is a factual completion task. A model can consistently answer the base and contextual questions differently simply because the questions are not the same type. Until the mapping is validated by human annotation, model-generated rationales, or an independent benchmark, the deviation defined in Definition 1 may reflect differences in question type rather than genuine preference inconsistency.
- [Section 4.3, Tables 2 and 3] All probabilities are empirical frequencies over 11 base-prompt versions and 4 to 24 contextual prompts, with no repeated sampling, no confidence intervals, and no significance tests. The overall absolute deviations (GPT 0.371, Gemini 0.355) and the per-category comparisons are therefore not distinguishable from sampling noise; a single row such as RP_4 with an absolute deviation of 1.000 can be based on very few prompts. The paper should report bootstrap confidence intervals, repeated sampling with temperature > 0, or another form of uncertainty quantification before claiming systematic differences between models.
- [Abstract and Section 4.4] The abstract claims evaluation on 'four mainstream LLMs,' but Section 4.1 lists three models, and Section 4.4 excludes Claude post hoc because 84% of its base-prompt responses are neutral. The headline that the phenomenon holds 'regardless of the preference categories and LLMs in the test' is therefore supported by only two models. Either the claim should be narrowed to GPT and Gemini, or Claude should be analyzed on the subset of categories where its stated preferences can be inferred, with a sensitivity analysis that accounts for the 84% neutrality rate.
- [Abstract, Section 4.3, and Appendix Table A1] The central claim describes a 'minor change in prompt format,' but the manipulations include substantive changes to decision-relevant parameters: the number of people sacrificed, the gender of the decision-maker, outcome probabilities, peer behavior, and gain/loss framing. These are content changes, not merely format changes. The finding is better characterized as context-sensitivity, and the current wording overstates what the experiments actually demonstrate.
- [Equation (7) and Appendix B.3] The KL divergence as computed depends on an arbitrary smoothing constant (epsilon = 0.001) that appears only in the appendix, and on the choice of base-10 logarithms. The paper does not report sensitivity to epsilon or to the logarithm base. Additionally, Appendix A states that neutral responses are 'discarded and counted as aligning with neither' principle; if neutrality rates differ between base and contextual prompts, the frequency-based probabilities and the resulting divergence can be driven by the handling of neutral responses rather than by preference shifts. This should be made explicit and analyzed.
minor comments (6)
- [Section 3.1, Equation (2)] The notation RP (CA, CB) = RP (C(P|Context)) is unclear because C(P|Context) is not defined; please rewrite the equation and define all symbols.
- [Tables 2 and 3] Rows CP_1 and EE_1 appear under the Miscellaneous category but are not explained in the text; please define all category abbreviations and row labels.
- [Abstract and Section 6] The abstract says 'four mainstream LLMs' while Section 4.1 lists three, and Section 6 contains the typo 'With in the three selected LLMs'; these inconsistencies should be corrected.
- [Section 4.3] The paper does not state the API sampling parameters (temperature, top_p, number of samples, access dates) used for the commercial models, which limits reproducibility.
- [Section 6] The speculation about future LLM deviations as 'hallmarks of consciousness or proto-conscious agency' is unsupported by the study and should be removed or explicitly labeled as speculation outside the paper's scope.
- [Dataset description] The abstract says the dataset can serve as a benchmark, but no repository link, license, or release details are provided; please add an availability statement.
Circularity Check
No significant circularity: the deviation metrics are direct measurements of response frequencies, not predictions derived from fitted parameters or self-cited results.
full rationale
The paper's derivation chain is self-contained and non-circular. Stated preference probabilities are computed as selection frequencies over the base prompt and 10 paraphrased variants, and revealed preference probabilities are computed over contextualized prompt variants; Equations (6) and (7) then define absolute deviation and KL divergence as distances between these measured distributions. No parameter is fitted to a subset of the data and then renamed as a prediction. The 50% dominance threshold and the epsilon smoothing factor (0.001) are transparent decision rules, not free parameters that force the reported deviations. The paper contains no load-bearing self-citations: all cited prior work is external and used for background or motivation, not to justify the central measurement. The Appendix A claim that 'each choice represents a semantic mapping to one of the principles previously tested' is an unvalidated construct-validity assumption, and the base and contextual prompts may indeed differ in question type (descriptive belief vs. personal decision); however, this is a concern about whether the metric measures what it claims, not a circular reduction in which the target result is embedded in the inputs or equations. The empirical finding that response frequencies shift across prompt formats is directly measured, so no circularity is present.
Assumptions & free parameters
free parameters (3)
- epsilon smoothing in KL denominator =
0.001
- dominance threshold =
50% (majority of 11 paraphrases)
- paraphrase count =
10 per base prompt
assumptions (4)
- domain assumption Response frequencies over 11 paraphrases estimate the model's probability of adopting a principle.
- domain assumption Choices in contextual prompts can be mapped to the same two principles as the base prompt.
- domain assumption Neutral responses can be discarded as aligning with neither principle.
- domain assumption LLM responses are deterministic enough for single-shot frequency estimation.
Cite this review
Pith. "Pith review of Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?." pith.science (2026). https://pith.science/paper/MY762G5A
@misc{pith2026250600751,
author = {Pith},
title = {Pith review of: Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?},
year = {2026},
howpublished = {\url{https://pith.science/paper/MY762G5A}},
note = {Machine review of arXiv:2506.00751}
}
read the original abstract
Recent advances in Large Language Models (LLMs) highlight the need to align their behaviors with human values. A critical, yet understudied, issue is the potential divergence between an LLM's stated preferences (its reported alignment with general principles) and its revealed preferences (inferred from decisions in contextualized scenarios). Such deviations raise fundamental concerns for the interpretability, trustworthiness, reasoning transparency, and ethical deployment of LLMs, particularly in high-stakes applications. This work formally defines and proposes a method to measure this preference deviation. We investigate how LLMs may activate different guiding principles in specific contexts, leading to choices that diverge from previously stated general principles. Our approach involves crafting a rich dataset of well-designed prompts as a series of forced binary choices and presenting them to LLMs. We compare LLM responses to general principle prompts stated preference with LLM responses to contextualized prompts revealed preference, using metrics like KL divergence to quantify the deviation. We repeat the analysis across different categories of preferences and on four mainstream LLMs and find that a minor change in prompt format can often pivot the preferred choice regardless of the preference categories and LLMs in the test. This prevalent phenomenon highlights the lack of understanding and control of the LLM decision-making competence. Our study will be crucial for integrating LLMs into services, especially those that interact directly with humans, where morality, fairness, and social responsibilities are crucial dimensions. Furthermore, identifying or being aware of such deviation will be critically important as LLMs are increasingly envisioned for autonomous agentic tasks where continuous human evaluation of all LLMs' intermediary decision-making steps is impossible.
Figures
Reference graph
Works this paper leans on
-
[1]
Base Prompt (Stated Preference): A forced binary-choice question that directly elicits the LLM’s stated preference. This prompt presents a generalized, abstract scenario and asks the model to choose between two principles or values, thereby making its normative reasoning explicit
-
[2]
You” in vr1 to “The other customer
Contextualized Prompt (Revealed Preference): A forced binary-choice question embed- ded in a more concrete setting. While the prompt does not mention the guiding principles directly, each choice represents a semantic mapping to one of the principles previously tested. The LLM’s selected option allows us to infer the latent guiding principle underlying its...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.