arxiv: 2605.08556 · v1 · submitted 2026-05-08 · 💻 cs.LG

Recognition: no theorem link

Can Revealed Preferences Clarify LLM Alignment and Steering?

Khurram Yamin , Jingjing Tang , Eric Horvitz , Bryan Wilder

Authors on Pith no claims yet

Pith reviewed 2026-05-12 02:05 UTC · model grok-4.3

classification 💻 cs.LG

keywords LLM alignmentrevealed preferencesdiscrete choice modelspreference elicitationmodel steeringmedical diagnosisAI safetydecision coherence

0 comments

The pith

Revealed preference analysis shows many LLMs act coherently yet fail to report or adopt user-specified goals.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a pipeline that elicits an LLM's probability estimates over unknowns and its binary choices in decision tasks, then fits a discrete choice model to recover the cost function that best explains those choices. This recovered cost function serves as a quantitative description of the model's implied preferences, which is compared against the model's verbal self-description and against the effects of steering prompts that instruct it to follow a different cost function. The method is tested on four medical diagnosis domains with multiple frontier and open-source models. The central finding is that models often display internal coherence in their choices but show clear gaps in accurately describing their own objectives and in shifting to user-directed policies. A reader would care because LLMs are already supporting high-stakes decisions where verbal assurances of alignment may not match actual behavior.

Core claim

We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply t

What carries the argument

The revealed-preference pipeline that elicits choice probabilities and decisions then fits a discrete choice model to recover the cost function rationalizing the observed behavior.

If this is right

Alignment evaluation can move beyond verbal self-reports to quantitative comparisons between revealed cost functions and intended objectives.
Prompt-based steering cannot be assumed to change the underlying decision policy even when the model appears to accept the instruction.
Medical diagnosis tasks provide a concrete domain where coherence can be measured against objective tradeoffs between diagnostic accuracy, cost, and patient risk.
The same pipeline can be reused across other decision domains to produce comparable measures of goal-directedness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If recovered cost functions differ systematically from training objectives, the method could help isolate which data or reward signals produce specific misalignments.
The approach might be extended to measure how fine-tuning or RLHF alters the gap between revealed and stated preferences.
Repeated application over time could track whether model updates improve or degrade steerability and self-reporting fidelity.

Load-bearing premise

The discrete choice model fitted to elicited probabilities and binary choices accurately recovers the LLM's true underlying preference structure rather than merely describing surface-level output patterns.

What would settle it

Apply the pipeline to a model on a training set of diagnosis tasks, derive its cost function, then test whether that same cost function predicts the model's choices on a new held-out set of similar tasks with accuracy no better than chance.

Figures

Figures reproduced from arXiv: 2605.08556 by Bryan Wilder, Eric Horvitz, Jingjing Tang, Khurram Yamin.

**Figure 2.** Figure 2: Response of implied utility ratios to explicit cost-function prompting for the diagnostic [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Figure 3: Counterfactual predictions versus realized steering benefits. The x-axis shows the counterfactual-predicted percent loss reduction; the y-axis shows the realized percent loss reduction under the corresponding prompt intervention. Colors indicate domains, marker shapes indicate models, and numbers indicate benchmark cost functions. (a) Replacing the baseline implied cost function with the benchmark cost fun… view at source ↗

**Figure 4.** Figure 4: Response of baseline implied utility ratios to explicit cost-function prompting for both the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

**Figure 5.** Figure 5: Sensitivity of implied loss-function ratios to Gaussian noise in elicited beliefs. We add independent Gaussian noise to the elicited probabilities, clip the perturbed beliefs to [0, 1], and re-estimate the implied loss function using the same MLE pipeline as in the main text. The two panels report the median absolute percent change in the implied FN/FP ratio (left) and Defer/FP ratio (right) as the noise s… view at source ↗

read the original abstract

LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The revealed-preference pipeline for auditing LLMs is a solid idea in principle, but the paper's lack of detail on methods leaves the key claims hard to evaluate.

read the letter

This paper puts forward a pipeline for using revealed preferences to understand LLM alignment. They get the model to output probabilities over outcomes and make choices in medical diagnosis tasks, fit a discrete choice model to recover the cost function that explains those choices, and then use that to check coherence, whether verbal descriptions match the revealed policy, and if prompts can steer the policy. The new part is this empirical application to LLMs, turning economic methods into a tool for auditing decision tradeoffs. It does well by focusing on actual choices under uncertainty instead of just asking the model what it values. The soft spots center on the execution. There is no information provided on how the discrete choice model is specified or fitted, what the data volumes are, or any tests for robustness. This makes it tough to tell if the reported weaknesses in reporting and adopting preferences are real or artifacts of the elicitation. The assumption that the fitted cost function reflects a stable internal structure rather than prompt-driven output patterns needs more support, like showing invariance to rephrasing or performance on new choice sets. Readers working on alignment evaluation and safety would get something from the framework. It is worth sending for peer review so the authors can fill in the details and address whether the recovered preferences hold up.

Referee Report

3 major / 2 minor

Summary. The paper proposes an empirical pipeline to recover implied preferences of LLMs: elicit probability distributions over outcomes and binary choices in medical diagnosis tasks, then fit a discrete choice model to obtain a cost function that rationalizes the observed decisions. This recovered cost function is used to quantify internal coherence of the model's choices, the match between verbalized objectives and revealed policy, and the reliability of prompting to steer the policy toward a user-specified cost function. The pipeline is applied across four medical diagnosis domains and multiple frontier and open-source models, with the main finding that models exhibit nontrivial internal coherence yet show significant weaknesses in faithfully reporting or adopting preferences under user direction.

Significance. If the recovered cost function is shown to represent a stable decision policy rather than surface patterns, the approach supplies a quantitative, revealed-preference framework for evaluating alignment properties that go beyond factual accuracy, which is directly relevant to high-stakes decision support. The empirical scope across domains and models provides concrete, falsifiable measurements of coherence and steerability that could serve as a benchmark for future work.

major comments (3)

[Methods / Evaluation pipeline] The central claims rest on the assumption that the fitted discrete choice model recovers the LLM's underlying preference structure. Because LLMs generate tokens via next-token prediction, the recovered cost function may instead capture prompt-induced correlations or training-data regularities that are consistent only within the elicitation format. No independent validation (e.g., generalization to held-out choice sets or invariance under prompt rephrasing) is reported, which directly affects the interpretation of the coherence, verbalization, and steering results.
[Results] The abstract and results sections state that 'many models have a nontrivial degree of internal coherence' and 'significant weaknesses,' yet no quantitative thresholds, statistical tests, sample sizes, or robustness checks are supplied for these characterizations. Without these details it is impossible to determine whether the data support the reported findings or whether the coherence metric is merely descriptive of surface behavior.
[Methods] The fitting procedure for the discrete choice model (model specification, loss function, optimization method, and handling of probability elicitations) is not described with sufficient precision to allow reproduction or to assess whether the recovered cost function is unique or sensitive to elicitation format.

minor comments (2)

[Abstract] The four medical diagnosis domains are referenced but not named in the abstract; listing them would improve immediate clarity.
[Introduction] Notation for the cost function and random-utility model should be introduced earlier and used consistently across sections.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive comments, which have helped us identify areas for improvement in the manuscript. We address each major comment below, indicating the revisions we plan to make.

read point-by-point responses

Referee: [Methods / Evaluation pipeline] The central claims rest on the assumption that the fitted discrete choice model recovers the LLM's underlying preference structure. Because LLMs generate tokens via next-token prediction, the recovered cost function may instead capture prompt-induced correlations or training-data regularities that are consistent only within the elicitation format. No independent validation (e.g., generalization to held-out choice sets or invariance under prompt rephrasing) is reported, which directly affects the interpretation of the coherence, verbalization, and steering results.

Authors: We agree that validating whether the recovered cost function reflects a stable underlying preference rather than elicitation-specific patterns is important for the robustness of our claims. While our current analysis focuses on coherence within the elicited tasks, we will add new experiments in the revised manuscript to test generalization to held-out choice sets and invariance under different prompt phrasings. This will provide evidence on the stability of the recovered preferences and strengthen the interpretation of our coherence, verbalization, and steering results. revision: partial
Referee: [Results] The abstract and results sections state that 'many models have a nontrivial degree of internal coherence' and 'significant weaknesses,' yet no quantitative thresholds, statistical tests, sample sizes, or robustness checks are supplied for these characterizations. Without these details it is impossible to determine whether the data support the reported findings or whether the coherence metric is merely descriptive of surface behavior.

Authors: We acknowledge the need for more rigorous quantitative support for our characterizations. In the revised manuscript, we will define explicit quantitative thresholds for 'nontrivial coherence' based on the distribution of coherence metrics across models, include statistical tests (such as t-tests or Wilcoxon tests for comparisons), report exact sample sizes for all experiments, and add robustness checks including variations in elicitation parameters. These additions will allow readers to better assess the strength of the evidence for our findings. revision: yes
Referee: [Methods] The fitting procedure for the discrete choice model (model specification, loss function, optimization method, and handling of probability elicitations) is not described with sufficient precision to allow reproduction or to assess whether the recovered cost function is unique or sensitive to elicitation format.

Authors: We apologize for the lack of sufficient detail in the methods section regarding the discrete choice model fitting. In the revised version, we will expand this section to include the exact model specification (e.g., the form of the cost function and choice probabilities), the loss function used for fitting (negative log-likelihood of observed choices), the optimization procedure (including any regularization or initialization), and how the elicited probability distributions are integrated into the likelihood. We will also report analyses on the uniqueness of the recovered cost function and its sensitivity to changes in the elicitation format. revision: yes

Circularity Check

0 steps flagged

No significant circularity; standard revealed-preference fitting applied to LLM outputs

full rationale

The paper's pipeline elicits probability distributions and binary choices from LLMs over medical diagnosis tasks, then fits a discrete choice model to recover an implied cost function. This is a conventional empirical estimation step whose parameters are determined by the observed data rather than presupposing the target conclusions about internal coherence, verbalization mismatch, or prompt steerability. Those conclusions are obtained by separate comparisons between the fitted model, the model's own verbalized objectives, and its responses under user-specified prompts. No equations reduce to their inputs by construction, no uniqueness theorems are imported from self-citations, and no ansatz is smuggled in. The approach remains falsifiable against new choice sets and prompt variants, making the derivation self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The central claim rests on the assumption that a fitted discrete-choice model faithfully captures an LLM's internal preferences; the abstract supplies no independent evidence or external benchmarks for this mapping.

pith-pipeline@v0.9.0 · 5470 in / 1076 out tokens · 36230 ms · 2026-05-12T02:05:45.809459+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 3 internal anchors

[1]

doi: 10.1126/science.adf6369. Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. A limited memory algorithm for bound constrained optimization.SIAM Journal on Scientific Computing, 16(5):1190–1208,

work page doi:10.1126/science.adf6369
[2]

Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu

doi: 10.1137/ 0916069. URLhttps://doi.org/10.1137/0916069. Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S., Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M., Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau, J., K...

work page doi:10.1137/0916069
[3]

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

URLhttps://arxiv.org/abs/2307.15217. Centers for Disease Control and Prevention. Cdc diabetes health indicators,

work page internal anchor Pith review arXiv
[4]

URLhttp://dx.doi

ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URLhttp://dx.doi.org/10.1038/s41586-025-09422-z. Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., and Rueckert, D. Evaluation and mitigation of the limitations of large language models in clinical decision-making.Nature Medicin...

work page doi:10.1038/s41586-025-09422-z
[5]

et al.: Evaluation and mitigation of the limitations of large language models in clinical decision-making

doi: 10.1038/s41591-024-03097-1. 10 Hensher, D. A., Rose, J. M., and Greene, W. H.Applied Choice Analysis. Cambridge University Press, 2 edition,

work page doi:10.1038/s41591-024-03097-1
[6]

Khalaf, H., Verdun, C

URLhttps://arxiv.org/abs/2406.05972. Khalaf, H., Verdun, C. M., Oesterling, A., Lakkaraju, H., and du Pin Calmon, F. Inference-time reward hacking in large language models,

work page arXiv
[7]

Inference-time reward hacking in large language models.arXiv preprint arXiv:2506.19248,

URLhttps://arxiv.org/abs/2506.19248. Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., and Bernardo, D. Limitations of large language models in clinical problem-solving arising from inflexible reasoning, Nov

work page arXiv
[8]

Liu, R., Geng, J., Peterson, J

URLhttps://arxiv.org/abs/2509.25369. Liu, R., Geng, J., Peterson, J. C., Sucholutsky, I., and Griffiths, T. L. Large language models assume people are more rational than we really are.arXiv preprint arXiv:2406.17055,

work page arXiv
[9]

McFadden, D

URLhttps://arxiv.org/abs/2502.08640. McFadden, D. Conditional logit analysis of qualitative choice behavior.Frontiers in Econometrics, pp. 105–142,

work page arXiv
[10]

Paruchuri, A., Garrison, J., Liao, S., Hernandez, J., Sunshine, J., Althoff, T., Liu, X., and McDuff, D

URLhttps://arxiv.org/abs/2406.01168. Paruchuri, A., Garrison, J., Liao, S., Hernandez, J., Sunshine, J., Althoff, T., Liu, X., and McDuff, D. What are the odds? language models are capable of probabilistic reasoning,

work page arXiv
[11]

Pfohl, S

URL https://arxiv.org/abs/2406.12830. Pfohl, S. R., Cole-Lewis, H., Sayres, R., Neal, D., Asiedu, M., Dieng, A., Tomasev, N., Rashid, Q. M., Azizi, S., Rostamzadeh, N., McCoy, L. G., Celi, L. A., Liu, Y ., Schaekermann, M., Walton, A., Parrish, A., Nagpal, C., Singh, P., Dewitt, A., Mansfield, P., Prakash, S., Heller, K., Karthikesalingam, A., Semturs, C....

work page arXiv
[12]

doi: 10.1038/ s41591-024-03258-2

ISSN 1546-170X. doi: 10.1038/ s41591-024-03258-2. URLhttp://dx.doi.org/10.1038/s41591-024-03258-2. Samway, K., Kleiman-Weiner, M., Piedrahita, D. G., Mihalcea, R., Schölkopf, B., and Jin, Z. Are language models consequentialist or deontological moral reasoners? In Christodoulopoulos, C., Chakraborty, T., Rose, C., and Peng, V . (eds.),Proceedings of the 2...

work page doi:10.1038/s41591-024-03258-2 2025
[13]

In: NAACL (Long Papers)

Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025. emnlp-main.1563. URLhttps://aclanthology.org/2025.emnlp-main.1563/. Singh, A., Fry, A., Perelman, A., Tart, A., Ganesh, A., El-Kishky, A., McLaughlin, A., Low, A., Ostrow, A., Ananthram, A., Nathan, A., Luo, A., Helyar, A., Madry, A., Efremov, A., Spyra, A., Baker-Wh...

work page doi:10.18653/v1/2025 2025
[14]

URLhttps://arxiv.org/abs/2601.03267. Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., Rashid, Q., Schaekermann, M., Wang, A., Dash, D., Chen, J., Shah, N., Lachgar, S., Mansfield, P., Prakash, S., Green, B., Dominowska, E., Agüera y Arcas, B., Tomašev, N., Liu, Y ., Wong, R., Se...

work page internal anchor Pith review Pith/arXiv arXiv
[15]

Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H

doi: 10.1038/s41591-024-03423-7. URLhttps://doi.org/10.1038/s41591-024-03423-7. Slama, K., Souly, A., Bansal, D., Davidson, H., Summerfield, C., and Luettgau, L. When do llm preferences predict downstream behavior?,

work page doi:10.1038/s41591-024-03423-7
[16]

Train, K

URL https://arxiv.org/abs/2602.18971. Train, K. E.Discrete Choice Methods with Simulation. Cambridge University Press, 2nd edition,

work page arXiv
[17]

URL https://doi.org/10.1371/journal.pdig.0000899

doi: 10.1371/journal.pdig.0000899. URL https://doi.org/10.1371/journal.pdig.0000899. Xiao, F. and Wang, X. X. Evaluating the ability of large language models to predict human social decisions.Scientific Reports, 15(1):32290,

work page doi:10.1371/journal.pdig.0000899
[18]

When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs

URL https://arxiv.org/abs/2602.06286. Zhu, J.-Q., Yan, H., and Griffiths, T. L. Steering risk preferences in large language models by aligning behavioral and neural representations,

work page internal anchor Pith review Pith/arXiv arXiv
[19]

URLhttps://arxiv.org/abs/2505.11615. 13 Technical appendices and supplementary material A Belief Validity A.1 Belief Validity Comparison We first ask which elicited probability object is more behaviorally useful for decisions made under cost-function prompting: probabilities elicited from the standard prompt or from a prompt that includes the loss functio...

work page arXiv 2078
[20]

is male,

Example format: No: 0.30 Yes: 0.70 C.2 Decision Elicitation Prompts C.2.1 Decision Prompt A: Without Loss Function This prompt asks the model to make a diagnostic decision without specifying an explicit loss function. The model is asked both (1) whether it feels confident enough to make a decision, allowing for deferral, and (2) what decision it would mak...

work page 2017