REVIEW 3 major objections 5 minor 40 references
Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices
T0 review · 3 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read The paper shows that LLM travel assistants have measurable, dollar-valued preferences for hotel attributes—internally coherent yet misaligned with human benchmarks—and that these values swing with prompts indicating cheap or expensive prefe
desk verdict Solid new application of WTP econometrics to LLM choices, but the headline WTP numbers wobble under order-switch; the qualitative prompt effects survive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the discrete choice experiment read through a multinomial logit model. Each of 240 dilemmas offers two hotel rooms described by six attributes plus a nightly price; the LLM's forced A/B choice is treated as the argmax of a random utility function U(c) = ε_c + α_c + β_c·x_c, with choice probability proportional to exp(U_c). Willingness to pay for attribute k is the ratio WTP_k = β_k·σ_price / (β_price·σ_k), which converts standardized logit coefficients into Hong Kong dollars. This machinery is what turns bare 'A or B' answers into a per-attribute monetary price, making LLM preferences directly comparable to human WTP benchmarks from the economics literature, and maki
What would settle it
Run the same 240 dilemmas under several shuffled option orders at temperature 0 and re-estimate the WTP each time; if the dollar values move as much as in the paper's own order-swap check (e.g., the view's WTP for one model drops from 511 to 193 HK$), the reported valuations are presentation artifacts rather than stable preferences. Alternatively, hold the prompt fixed and swap in a fresh, non-overlapping set of hotel dilemmas; stable WTP across dilemmas would support meaningfulness, while large swings would refute it.
Extended reading notes
Core claim
The paper's central discovery is that an LLM's choices in paired hotel dilemmas can be modelled as random utility maximization, yielding willingness-to-pay estimates in Hong Kong dollars per attribute. The three large models tested show internally coherent preferences—positive utility for desirable attributes, negative for price—yet their WTP values systematically miss the human benchmark: club access is overvalued two- to fourfold, the view undervalued, and overall WTP runs high. Prompt conditioning shifts these values: cheap-example prompts pull estimates toward human levels, while expensive-example prompts or a business persona inflate them, sometimes by factors above 40. The paper conclu
Load-bearing premise
The load-bearing premise is that a model's 240 deterministic choices behave like independent, identically distributed random draws from a stable utility distribution, even though the paper's own order-switching tests show that choice position influences responses.
Editorial extensions
If this is right
- LLM-based travel assistants will, without careful prompt design, systematically recommend options that exceed what a typical user would pay, since model WTP runs high on average.
- Attribute descriptions double as price controls: shortening the club-access clause from a detailed sentence to a short phrase cut the estimated WTP by more than half, so wording has a measurable monetary effect.
- In-context examples of cheap prior choices are the most effective tested lever for aligning LLM valuations with human ones; expensive examples actively misalign them.
- Persona prompting can produce extreme, even degenerate, behavior—one model under a 'student' persona simply always chose the cheapest room, making WTP estimation impossible; another under a 'business' persona produced a positive price coefficient, implying it 'prefers' higher prices.
- Model scale is a real constraint: smaller models exhibited order bias and illogical price coefficients, and the authors exclude them from WTP analysis, implying they are unsuitable for subjective decision support.
Reading between the lines
- If WTP estimates are as prompt-sensitive as reported, they cannot be treated as stable model properties; an immediate testable extension is to randomize attribute descriptions across repeated runs and measure how much of the variance in WTP is explained by lexical content alone.
- The same multinomial-logit pipeline transfers to other subjective choice domains—insurance plans, subscription bundles, vacation itineraries—where it could yield dollar-valued readouts of an LLM's implicit priorities, provided fresh human benchmarks are collected for each domain.
- The paper's human comparison rests on a single 2015 survey; because preferences drift over time, part of the measured model–human gap may be benchmark aging rather than model bias. A contemporaneous human sample would separate these two effects.
- The residual order bias the paper documents in larger models suggests the iid error assumption is violated; fitting the same model to choices collected under several shuffled presentation orders and checking whether WTP estimates fall within a common confidence band would test whether the reported dollar values are stable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an economic framework for evaluating LLM decision-making: it presents LLMs with paired hotel-room choice dilemmas, fits multinomial logit models to the resulting binary choices, and converts the estimated coefficients into willingness-to-pay (WTP) values for attributes such as view, floor, club access, minibar, smartphone, and cancellation. These WTP estimates are compared with human benchmark values from Masiero et al. (2015). The paper studies a baseline condition and several realistic conditions involving in-context learning, persona prompting, and their combination, using three larger LLMs (Llama 3.3 70B, GPT-4o, Gemini-3-Pro). The main claims are that meaningful WTP values can be derived for larger LLMs; that these models show systematic attribute-level deviations from human valuations, notably overvaluing club access and undervaluing view/floor; that LLMs tend to overestimate human WTP overall; and that conditioning on cheap prior choices moves estimates closer to human benchmarks, while expensive priors and business personas amplify overestimation.
Significance. If the central claims hold, the paper makes a useful methodological contribution by importing discrete-choice economics into LLM evaluation: WTP provides an interpretable, monetary scale on which to compare model and human preferences, and the open-source codebase supports replication. The comparison against an external human benchmark is a strength, as is the inclusion of order-switch and currency robustness checks. The qualitative finding that prompt conditioning can systematically shift valuations in a measurable direction is practically relevant for LLM-based travel and purchasing assistants. However, the validity of the quantitative WTP estimates rests on the multinomial logit assumption of iid errors, and the paper's own order-switch experiments show that this assumption is violated. Because the point estimates and significance tests are load-bearing for the claim that 'meaningful WTP values' can be derived, the paper needs substantial revision before the central claim is established.
major comments (3)
- [Section 3.4 / Appendix A.1] The paper's own robustness check contradicts the identifying assumption. Section 3.4 states that the iid error assumption 'would be violated if there is order bias present.' Table A.9 shows that when A and B are swapped, the alternative-specific constant becomes statistically significant for GPT-4o and Gemini-3-Pro, and remains significant for Llama 3.3 70B, so order bias is present in all three models. The order switch also changes WTP point estimates materially: for example, Llama view goes from 511.32 to 192.81 HKD, GPT-4o view from 115.66 to 364.91, and Llama floor from 8.29 to 27.52 (Table 4 vs. Table A.10). The last case crosses the human benchmark of 22 HKD and reverses the Section 4.1 claim that all models undervalue floor. Averaging over two order runs (Figure A.8) does not repair the iid violation, because position effects need not enter linearly. The authors should re-estimate
- [Section 3.3 / Section 3.4, Eq. (2)] The multinomial logit model in Eq. (2) is a stochastic model, but the experiments use temperature 0, yielding deterministic responses. Each of the 240 choice outcomes is a deterministic function of the prompt; there is no sampling from a choice distribution that would justify the likelihood-based standard errors and significance tests. The only stochastic variation is across choice sets, and the fitted model is at best a descriptive approximation of a deterministic decision rule. This matters for the claim that 'meaningful WTP values can be derived for larger LLMs': the reported p-values and pseudo-R2 do not provide the usual inferential guarantees. The authors should either obtain genuinely stochastic choices (e.g., temperature > 0 or multiple seeds) and model those, or explicitly reframe all WTP claims as descriptive of the deterministic input-output mapping and remove significance-bas
- [Section 3.3 and Section 5.1] Four smaller models were excluded post hoc because they exhibited low pseudo-R2, order bias, or illogical WTP signs (e.g., positive price coefficient). These are exactly the phenomena studied in the paper, so the selection of the three 'larger' models is outcome-dependent. The generalization that 'larger LLMs' display systematic deviations while smaller models are unsuitable is therefore not established; it is conditional on models for which the MNL estimation happens to produce economically interpretable results. The authors should either report the smaller-model results as a sensitivity analysis, or clearly restrict the conclusions to the three selected models and justify the selection criteria prospectively.
minor comments (5)
- [Section 3.4] The text says the errors are 'independent from each other and normally distributed,' but the multinomial logit model in Eq. (2) assumes iid extreme-value (Gumbel) errors. Please correct this inconsistency.
- [Table 3 / Table A.9] The caption says the pseudo-R2 values are averaged over two order runs, but the coefficients are from the first run only. This should be stated in the main text and ideally the second-run coefficients or averaged estimates should be reported in parallel to avoid cherry-picking impressions.
- [Section 4.2.2] For Gemini-3-Pro under the student persona, perfect separation is described as a 'strong result,' while under the business persona a positive price coefficient leads to WTP estimates being withheld. Both are instances of extreme persona effects and should be treated symmetrically, with explicit discussion of what each implies for the practicality of persona-based conditioning.
- [Appendix A.2] Figure A.9 caption text contains rendering artifacts ('/uni00000039/uni0000004c/...') in the provided manuscript. The final version should be checked so that figures and captions render correctly.
- [Various tables of WTP values] WTP estimates are ratios of coefficients, but no confidence intervals or standard errors are reported for them. Even if the coefficient-level inference were valid, the WTP ratios may have wide intervals. Reporting at least bootstrap or delta-method intervals would make the magnitudes of the reported differences more interpretable.
Circularity Check
No significant circularity: WTP estimates are derived from LLM choices and benchmarked against an external human study; the only self-citations are non-load-bearing design choices.
full rationale
The paper's central claim is that multinomial logit models fitted to LLM choices yield interpretable WTP estimates that can be compared with human benchmarks. This derivation is self-contained: WTP is a closed-form function of the estimated MNL coefficients (Eq. 3), the choices are generated from the LLM prompts, and the human benchmark values come from the independent study by Masiero et al. [9]. No parameter is fitted to the benchmark and then renamed as a prediction; the comparison is made after estimation. The only self-citations are [5], used for the prompt template, and [8], cited in the context of persona prompting. These influence the experimental setup but do not constitute the derivation of WTP, and removing them would not change the estimation logic. The paper itself flags a real limitation in Section 3.4: the iid-error assumption is violated if order bias exists, and Appendix A.1 confirms that order bias persists in the larger models. That is a statistical validity concern about the stability and significance of the WTP estimates, not a circularity concern: the estimates are not defined in terms of the benchmark or produced by fitting the target conclusion. Consequently, no specific circular reduction can be exhibited, and the appropriate score is low, reflecting only the minor, non-load-bearing self-citations.
Assumptions & free parameters
assumptions (5)
- domain assumption LLM choices are generated by a random utility model with logistic errors (multinomial logit).
- domain assumption The 240 responses per condition are independent draws with iid normally distributed errors despite temperature 0.
- domain assumption Masiero et al. (2015) human WTP values remain the correct benchmark for 2025 preferences after CPI adjustment.
- domain assumption Attribute descriptions in the prompt preserve the same trade-offs as the original human choice experiment.
- standard math Standardized-coefficient rescaling yields WTP in monetary units.
Cite this review
Pith. "Pith review of Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices." pith.science (2026). https://pith.science/paper/KK2BGV3C
@misc{pith2026260209802,
author = {Pith},
title = {Pith review of: Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices},
year = {2026},
howpublished = {\url{https://pith.science/paper/KK2BGV3C}},
note = {Machine review of arXiv:2602.09802}
}
read the original abstract
As Large Language Models (LLMs) are increasingly deployed in applications such as travel assistance and purchasing support, they are often required to make subjective choices on behalf of users in settings where no objectively correct answer exists. We study LLM decision-making in a travel-assistant context by presenting models with choice dilemmas and analyzing their responses using multinomial logit models to derive implied willingness to pay (WTP) estimates. These WTP values are subsequently compared to human benchmark values from the economics literature. In addition to a baseline setting, we examine how model behavior changes under more realistic conditions, including the provision of information about users' past choices and persona-based prompting. Our results show that while meaningful WTP values can be derived for larger LLMs, they also display systematic deviations at the attribute level. Additionally, they tend to overestimate human WTP overall, particularly when expensive options or business-oriented personas are introduced. Conditioning models on prior preferences for cheaper options yields valuations that are closer to human benchmarks. Overall, our findings highlight both the potential and the limitations of using LLMs for subjective decision support and underscore the importance of careful model selection, prompt design, and user representation when deploying such systems in practice.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Gabriel, G
I. Gabriel, G. Keeling, A. Manzini, J. Evans, We need a new ethics for a world of ai agents, Nature 644 (8075) (2025) 38–40. 31 /uni00000039/uni0000004c/uni00000048/uni0000005a/uni00000029/uni0000004f/uni00000052/uni00000052/uni00000055/uni00000024/uni00000046/uni00000046/uni00000048/uni00000056/uni00000056 /uni00000026/uni0000004f/uni00000058/uni00000045...
2025
-
[2]
Jiang, A
Y. Jiang, A. Ng, Tech Overview - Stanford Agentic Reviewer — paper- review.ai, https://paperreview.ai, [Accessed 25-11-2025] (2025)
2025
-
[3]
OpenAI, Buy it in ChatGPT: Instant Checkout and the Agentic Com- merce Protocol, https://openai.com/en-EN/index/buy-it-in-chatgpt/, [Accessed 25-11-2025] (2025)
2025
-
[4]
OpenAI, Booking.com and OpenAI personalize travel at scale, https://openai.com/en-EN/index/booking-com/, [Accessed 25-11-2025] (2025)
2025
- [5]
-
[6]
P. Röttger, V. Hofmann, V. Pyatkin, M. Hinck, H. Kirk, H. Schuetze, D. Hovy, Political compass or spinning arrow? towards more mean- ingful evaluations for values and opinions in large language models, in: 32 L.-W. Ku, A. Martins, V. Srikumar (Eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long ...
-
[7]
W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, Y. Deng, Wildchat: 1m chatGPT interaction logs in the wild, in: The Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=Bl8u7ZRlbM
2024
-
[8]
M. Reusens, B. Baesens, D. Jurgens, Are economists always more introverted? analyzing consistency in persona-assigned LLMs, in: C. Christodoulopoulos, T. Chakraborty, C. Rose, V. Peng (Eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2025, Association for Computational Linguistics, Suzhou, China, 2025, pp. 11268–11287. doi:10.1865...
Show all 40 references
-
[9]
Masiero, C
L. Masiero, C. Y. Heo, B. Pan, Determining guests’ willingness to pay for hotel room attributes with a discrete choice model, In- ternational Journal of Hospitality Management 49 (2015) 117–124. doi:https://doi.org/10.1016/j.ijhm.2015.06.001. URLhttps://www.sciencedirect.com/s...
2015 doi
-
[10]
M. O. Rieger, M. Wang, T. Hens, Universal time preference, PLoS One 16 (2) (2021) e0245692
2021
-
[11]
M. Wang, M. O. Rieger, T. Hens, How time preferences differ: Evidence from 53 countries, Journal of Economic Psychology 52 (2016) 115–135. doi:https://doi.org/10.1016/j.joep.2015.12.001. URLhttps://www.sciencedirect.com/science/article/pii/ S0167487015001439
2016 doi
-
[12]
Dittrich, K
M. Dittrich, K. Leipold, Gender differences in time preferences, Economics Letters 122 (3) (2014) 413–415. doi:https://doi.org/10.1016/j.econlet.2014.01.002. URLhttps://www.sciencedirect.com/science/article/pii/ S0165176514000032 33
2014 doi
-
[13]
K. H. Kang, L. Stein, C. Y. Heo, S. Lee, Consumers’ willingness to pay for green initiatives of the hotel industry, International journal of hospitality management 31 (2) (2012) 564–572
2012
-
[14]
Scholz, V
M. Scholz, V. Dorner, M. Franz, O. Hinz, Measuring con- sumers’ willingness to pay with utility-based recommenda- tion systems, Decision Support Systems 72 (2015) 60–71. doi:https://doi.org/10.1016/j.dss.2015.02.006. URLhttps://www.sciencedirect.com/science/article/pii/ S01679...
2015 doi
-
[15]
P. A. Van Lange, J. Joireman, C. D. Parks, E. Van Dijk, The psy- chology of social dilemmas: A review, Organizational Behavior and Human Decision Processes 120 (2) (2013) 125–141, social Dilemmas. doi:https://doi.org/10.1016/j.obhdp.2012.11.003. URLhttps://www.sciencedirect.co...
2013 doi
-
[16]
Lisciandra, The role of psychology in behavioral economics: The case of social preferences, Studies in History and Philosophy of Science Part A 72 (2018) 11–21
C. Lisciandra, The role of psychology in behavioral economics: The case of social preferences, Studies in History and Philosophy of Science Part A 72 (2018) 11–21. doi:https://doi.org/10.1016/j.shpsa.2018.01.010. URLhttps://www.sciencedirect.com/science/article/pii/ S0039368118300244
2018 doi
-
[17]
de Andrés Calle, J
R. de Andrés Calle, J. Cascón, T. González-Arteaga, Preferences stabil- ity: A measure of preferences changes over time, Decision Support Sys- tems 129 (2020) 113169. doi:https://doi.org/10.1016/j.dss.2019.113169. URLhttps://www.sciencedirect.com/science/article/pii/ S0167923619301988
2020
-
[18]
F. Zhou, Y. Jiang, Y. Qian, Y. Liu, Y. Chai, Product consumptions meet reviews: Inferring consumer preferences by an explainable ma- chine learning approach, Decision Support Systems 177 (2024) 114088. doi:https://doi.org/10.1016/j.dss.2023.114088. URLhttps://www.sciencedirect...
2024
-
[19]
J. Ji, Y. Chen, M. Jin, W. Xu, W. Hua, Y. Zhang, Moralbench: Moral evaluation of llms, ACM SIGKDD Explorations Newsletter 27 (1) (2025) 62–71. 34
2025
-
[20]
J. Jia, Z. Yuan, J. Pan, P. E. McNamara, D. Chen, Decision-making be- havior evaluation framework for llms under uncertain context, Advances in neural information processing systems 37 (2024) 113360–113382
2024
-
[21]
A. Goli, A. Singh, Frontiers: can large language models capture human preferences?, Marketing Science 43 (4) (2024) 709–722
2024
-
[22]
R. Liu, J. Geng, J. C. Peterson, I. Sucholutsky, T. L. Griffiths, Large language models assume people are more rational than we really are, in: The Thirteenth International Conference on Learning Representations, 2025
2025
-
[23]
Seror, The moral mind (s) of large language models, arXiv preprint arXiv:2412.04476 (2024)
A. Seror, The moral mind (s) of large language models, arXiv preprint arXiv:2412.04476 (2024)
2024 arXiv
-
[24]
Y. Chen, T. X. Liu, Y. Shan, S. Zhong, The emergence of economic rationality of GPT., Proceedings of the National Academy of Sciences 120 (51) (2023) e2316205120
2023
-
[25]
Y. Feng, V. Choudhary, Y. R. Shrestha, Noise, adaptation, and strat- egy: Assessing llm fidelity in decision-making, in: Proceedings of the 2025 Conference on Empirical Methods in Natural Language Process- ing, 2025, pp. 7704–7717
2025
-
[26]
Jiang, X
H. Jiang, X. Zhang, X. Cao, C. Breazeal, D. Roy, J. Kabbara, Per- sonaLLM: Investigating the ability of large language models to express personality traits, in: K. Duh, H. Gomez, S. Bethard (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024, Asso- ci...
2024 doi
-
[27]
Santurkar, E
S. Santurkar, E. Durmus, F. Ladhak, C. Lee, P. Liang, T. Hashimoto, Whose opinions do language models reflect?, in: International Confer- ence on Machine Learning, PMLR, 2023, pp. 29971–30004
2023
-
[28]
C. Yu, Q. Tan, H. Lu, J. Gao, X. Yang, Y. Wang, Y. Wu, E. Vinitsky, Icpl: Few-shot in-context preference learning via llms, arXiv preprint arXiv:2410.17233 (2024). 35
2024 arXiv
-
[29]
Aroca-Ouellette, N
S. Aroca-Ouellette, N. Mackraz, B.-J. Theobald, K. Metcalf, Aligning llms by predicting preferences from user writing samples, in: Forty- second International Conference on Machine Learning, 2025
2025
-
[30]
Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, B. Chang, X. Sun, L. Li, Z. Sui, A survey on in-context learning, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, As- ...
2024 doi
-
[31]
Tonglet, M
J. Tonglet, M. Reusens, P. Borchert, B. Baesens, SEER : A knap- sack approach to exemplar selection for in-context HybridQA, in: H. Bouamor, J. Pino, K. Bali (Eds.), Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, Associ- ation for Com...
2023 doi
-
[32]
Greene, Discrete choice modeling, in: Palgrave handbook of econo- metrics: Volume 2: Applied econometrics, Springer, 2009, pp
W. Greene, Discrete choice modeling, in: Palgrave handbook of econo- metrics: Volume 2: Applied econometrics, Springer, 2009, pp. 473–556
2009
-
[33]
Arteaga, J
C. Arteaga, J. Park, P. B. Beeramoole, A. Paz, xlogit: An open-source python package for gpu-accelerated estimation of mixed logit models, Journal of Choice Modelling 42 (2022) 100339. doi:https://doi.org/10.1016/j.jocm.2021.100339
2022
-
[34]
Shmueli, To explain or to predict?, Statistical science (2010) 289–310
G. Shmueli, To explain or to predict?, Statistical science (2010) 289–310
2010
-
[35]
K. E. Train, Discrete choice methods with simulation, Cambridge uni- versity press, 2009
2009
-
[36]
Menard, Coefficients of determination for multiple logistic regression analysis, The American Statistician 54 (1) (2000) 17–24
S. Menard, Coefficients of determination for multiple logistic regression analysis, The American Statistician 54 (1) (2000) 17–24
2000
-
[37]
P. F. Van Erkel, P. Thijssen, The first one wins: Distilling the primacy effect, Electoral Studies 44 (2016) 245–254. 36
2016
-
[38]
Goldberg, Measuring and improving consistency in pretrained lan- guage models, Transactions of the Association for Computational Lin- guistics 9 (2021) 1012–1031
Y.Elazar, N.Kassner, S.Ravfogel, A.Ravichander, E.Hovy, H.Schütze, Y. Goldberg, Measuring and improving consistency in pretrained lan- guage models, Transactions of the Association for Computational Lin- guistics 9 (2021) 1012–1031
2021
-
[39]
K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Gong, X. Xie, Promptrobust: Towards evaluating the ro- bustness of large language models on adversarial prompts, in: Proceed- ings of the 1st ACM workshop on large AI systems and models with priv...
2023
-
[40]
Allouah, O
A. Allouah, O. Besbes, J. D. Figueroa, Y. Kanoria, A. Kumar, What is youraiagentbuying? evaluation, biases, modeldependence, &emerging implications for agentic e-commerce, arXiv preprint arXiv:2508.02630 (2025). 37
2025
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.