LLM-SAA: LLM-persona Generated Distributions for Decision-making

Jackie Baek; Will Ma; Yunhan Chen; Ziyu Chi

REVIEW 3 major objections 6 minor 16 references

LLM-SAA: LLM-persona Generated Distributions for Decision-making

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated distributions can guide good operational decisions when real data is scarce.

desk verdict Real methodological contribution on decision-aware evaluation of LLM-generated distributions, but the empirical low-data claim is overstated and lacks a contamination check. read the letter →

arxiv 2602.06357 v2 pith:3UZXJHOS submitted 2026-02-06 cs.LG

Jackie Baek , Yunhan Chen , Ziyu Chi , Will Ma This is my paper

classification cs.LG MSC 90C1568T50

keywords LLM-SAAsampleaverageapproximationcompetitiveratioassortmentoptimizationpricingnewsvendorLLM-generateddistributionsdecision-awareevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LLM-SAA, a pipeline that asks a large language model to generate an estimated distribution of customer preferences, willingness-to-pay, or demand, and then optimizes a decision against that distribution. The authors claim that in low-data regimes, decisions made this way consistently beat uninformed baselines and can rival choices based on a small number of real samples, across assortment, pricing, and newsvendor problems. They argue that standard distribution-distance metrics such as Wasserstein distance are misleading for this purpose, because what matters is the quality of the induced decision, not statistical closeness. To evaluate this, they define competitive-ratio metrics and show that the worst-case competitive ratio over problem parameters is efficiently computable for all three problems.

What carries the argument

The key machinery is the competitive-ratio metric C_theta(F, F_hat), which compares the true expected reward of the LLM-SAA action against the reward of the ground-truth optimal action, evaluated at the worst-case or average-case problem parameters theta. The paper's theoretical contribution is showing that the worst-case competitive ratio can be computed exactly for all three problems: for assortment, by enumerating action pairs and using a linear program to check feasibility of parameter constructions; for pricing, by exploiting that the optimal price lies on the support of the distribution and that the ratio is monotone in cost; for newsvendor, by analogous monotonicity of the ratio in th

What would settle it

If the same evaluation were run on datasets that postdate the LLM's training cutoff, and LLM-generated distributions no longer beat the random baseline in low-data regimes, the central claim would fail. A concrete check: test LLM-SAA on a demand or preference dataset collected after the LLM's knowledge cutoff, ensuring no contamination, and measure the worst-case competitive ratio against the same baselines.

Watch

Extended reading notes

Core claim

The central discovery is that optimizing a decision against an LLM-generated distribution—LLM-SAA—produces decisions that are practically useful, particularly when historical data is limited. Using real datasets (SUSHI rankings for assortment, Philippine cocoa willingness-to-pay for pricing, H&M demand for newsvendor), the authors measure competitive ratios that compare the expected reward of the LLM-SAA decision against the true optimal decision under the ground-truth distribution, swept over worst-case and average-case problem parameters. They find LLM-SAA consistently outperforms an uninformed random baseline and can be competitive with empirical distributions built from 5 to 30 real samp

Load-bearing premise

The evaluation assumes the LLM has not memorized the exact evaluation datasets, so its outputs reflect general world knowledge rather than leaked test data.

Editorial extensions

If this is right

In low-data regimes, a firm could use LLM-SAA as a substitute for expensive data collection, achieving decisions close to optimal without historical samples.
Decision-agnostic metrics like Wasserstein or Kolmogorov distance should not be used to validate LLM-generated distributions for operational tasks; decision-aware competitive ratios are the correct yardstick.
Persona steering can improve downstream decisions even when the LLM completely fails to imitate individuals, so distribution-level diversity matters more than individual-level fidelity.
The worst-case competitive ratio provides a principled, parameter-free way to benchmark different LLM prompting methods and models for decision-making tasks.
The same evaluation framework extends to other stochastic optimization problems beyond the three studied, wherever the optimal-action set can be characterized.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the LLM's advantage comes partly from capturing human-like reporting patterns (e.g., willingness-to-pay values ending in 0 or 5), LLM-SAA is likely to help most where human behavioral regularities are mirrored in the outcome distribution, and least where outcomes are driven by physical or mechanical processes without such regularities.
A testable extension: in a new product introduction with no historical sales, compare LLM-SAA pricing decisions against decisions from a small pilot sample; the paper's results suggest LLM-SAA will be closer to the pilot-optimal price than a random baseline for a broad range of cost parameters.
The worst-case competitive ratio could serve as a training signal: one could fine-tune an LLM to minimize this ratio (or maximize average-case CR) rather than to minimize statistical distance, directly optimizing the objective the paper argues matters.
The concordance between worst-case and average-case CR rankings hints that adversarial parameter selection may be a reliable proxy for typical performance, but this is only demonstrated across three problems and the evaluated instances.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, this is a genuine methodological contribution: it defines decision-aware competitive-ratio metrics for evaluating LLM-generated distributions and shows how to compute the worst-case ratio for assortment, pricing, and newsvendor. Second, the empirical headline is real but overstated — the newsvendor AvgCR results do not separate from Random, and the missing contamination check leaves the low-data claim unresolved.

What is actually new: the worst-case competitive-ratio computation over instance parameters for fixed F and Fhat is a clean idea, and the three algorithms in the appendices are credible. The lemmas are simple but correct: for pricing and newsvendor, monotonicity of the ratio reduces the search to breakpoint values; for assortment, the LP feasibility check handles cases where more than one alternative could be maximal. The paper also contributes a useful negative finding: decision-agnostic metrics like Wasserstein can rank methods opposite to their decision quality, and persona-steering improves decisions even when individual-level imitation fails. That last point is important for the LLM-simulation literature.

The empirical protocol is solid in many ways: multiple LLMs, multiple reward sequences, confidence intervals, 20 generations per method, and the code is public. The assortment and pricing results do show LLM-SAA beating Random on the CR metrics, and the finding that Random has the best Wasserstein distance while performing worst on decision quality is a striking cautionary tale.

Soft spots, in proportion. The newsvendor section concedes that all models overlap with Random on AvgCR, and Random is better under Kolmogorov. So Takeaway I — 'consistently outperforms uninformed baselines' — is too strong as written. It holds for assortment and pricing, not for newsvendor under the average-case metric. The paper should qualify that. Second, the contamination issue is real and unaddressed. SUSHI, the cocoa WTP data, and H&M are all public datasets likely in LLM training corpora. The few-shot settings make it worse by injecting same-population examples (or, for H&M, same-retailer reference items). A simple probe or a privately held-out dataset would resolve this. Without it, the low-data generalization — the paper's stated motivation — is not established. The decision-aware evaluation framework stands regardless of the empirical outcome; that is the part I would trust.

Who this is for: operations researchers and ML people working on LLM simulation for decisions, and anyone benchmarking LLM-generated distributions. It deserves a serious referee. I would send it out, and require a contamination analysis plus a softened Takeaway I before acceptance.

Referee Report

3 major / 6 minor

Summary. LLM-SAA, the paper's pipeline, uses an LLM to construct an estimated distribution Fhat for an uncertain outcome and then optimizes a decision under Fhat via sample-average approximation. The authors propose decision-aware evaluation metrics based on competitive ratios (worst-case and average-case over instance parameters) and develop algorithms to compute the worst-case competitive ratio for assortment optimization, pricing, and the newsvendor problem. Using GPT-4o, GPT-5-mini, Gemini 3 Flash, and Mistral Large 3, they evaluate LLM-generated distributions on real datasets—SUSHI preference rankings, Philippine cocoa willingness-to-pay, and H&M demand—against random and empirical-sample baselines under several generation methods (sampling, persona-sampling, batch generation, description). They report that LLM-SAA often outperforms uninformed random baselines on competitive-ratio metrics and can be competitive with small real samples, while decision-agnostic distances (Wasserstein, Kolmogorov) are misleading indicators of decision quality. They also find that persona-based steering improves aggregate decisions even though the LLM fails to imitate individuals accurately.

Significance. The paper addresses a timely and important question: whether LLM-generated distributions can support downstream operational decisions in low-data regimes. Its decision-aware evaluation framework—measuring competitive ratios rather than distributional distances—is a useful contribution, and the formulation of worst-case competitive ratios for fixed F and Fhat is novel. The empirical study is broad, covering multiple LLMs, multiple prompting strategies, and three real datasets, with public code and confidence intervals. If the theoretical claims hold, the paper provides both a methodology and practical evidence for LLM-SAA. However, the theoretical result for assortment is compromised by a flawed lemma, and the headline empirical claims are weakened by the absence of data-contamination checks and by exceptions in the newsvendor and high-cost pricing results.

major comments (3)

[Section 4, Lemma 1] Lemma 1's construction is invalid. In the proof, B=1 and sizes c_j are set to 0, 2, 1, and 1/n, so the budget equals the average size. For n=3, take a*={1,2} and ahat={2,3}, with |a*\ahat|=1. The total size is 1 + 1/3, so the budget is 4/9; a* has size 1 and is infeasible. Similar failures occur when a* is a subset of ahat (ahat can be infeasible). Consequently, condition (4) plus min{|a*\ahat|,|ahat\a*|}≤1 does not guarantee that (2) holds, and Algorithm 1 may certify infeasible pairs and return a value different from the true WorstCR. The paper's claim to compute the worst-case competitive ratio for assortment is therefore not established.
[Sections 4–6, few-shot settings] The empirical claim that LLM-SAA is useful 'for new products and evolving market conditions' assumes the LLM's outputs reflect general world knowledge rather than memorized training data. The few-shot prompts inject examples from the same datasets as the ground truths: SUSHI respondents outside F (§4), cocoa WTP respondents outside F (§5), and H&M reference items from the same retailer (§6). No contamination or memorization check is reported. Since SUSHI and the H&M Kaggle dataset are public and could appear in LLM training corpora, the measured advantages—especially in the few-shot settings—may be inflated by leakage. Please add a memorization probe (e.g., asking the LLM to reproduce specific records) or evaluate on a private held-out dataset, and clearly separate zero-shot from few-shot results in the headline claims.
[Section 6.1 and Table 4] Takeaway I ('LLM-SAA consistently outperforms uninformed baselines') is not supported by all results. In the newsvendor problem (Section 6.1, Figure 3), AvgCR confidence intervals for all four LLM models overlap with the Random baseline, as the text acknowledges. In pricing, Table 4 shows several LLM methods below Random for AvgCR in the [0,100] cost range (e.g., GPT-4o Sampling 0.61 vs Random 0.67; Mistral Large 3 Persona 0.46). The abstract's 'practically useful' claim and Takeaway I should be qualified to the regimes where the evidence is positive, or the paper should justify why these exceptions do not contradict 'consistently outperforms.'

minor comments (6)

[Section 4, after Lemma 1] 'Theorem 1' is referenced but no Theorem 1 appears in the text; it should likely be 'Lemma 1.'
[Section 4, AvgCR/Hard regime] The 'AvgCR' for the Hard regime (s_j = S_j) is computed on a single deterministic instance, not an average over a distribution of instances. Please relabel this metric or clarify that it is the competitive ratio for that specific instance.
[Appendix I, prompts] In the Batch-generation and Description prompts for pricing, the instructions contain 'Consider the sushi item descriptions,' apparently copy-pasted from the assortment prompt; these should refer to the chocolate products.
[Section 3.2 and Section 4] For assortment, the WorstCR is defined over sizes and budget while holding the reward vector (r_p) fixed; this restriction should be stated explicitly in the metric definition in Section 3.2, since it affects the interpretation of the worst case.
[Figure 1] The legend does not indicate whether the displayed LLM methods are zero-shot or few-shot. Please clarify, as the few-shot results are also reported in Table 1 and the contamination concern differs between the two settings.
[Section 3.1/5.1] The average-case parameter distributions (e.g., cost ranges Unif[0,32], Unif[0,66], Unif[0,100]; q~Unif[0.01,0.99]) are introduced without justification. A short discussion or sensitivity analysis would help readers assess robustness of the AvgCR comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: LLM-SAA is evaluated against held-out ground-truth distributions with no parameter fitted to the conclusion.

full rationale

The paper's central claim (Takeaway I) is an empirical evaluation of the LLM-SAA pipeline: an LLM-generated distribution Fhat is compared to a ground-truth F drawn from real datasets, and downstream decision quality is measured through competitive ratios defined in Section 3.2. These ratios are computed from F and Fhat directly; no free parameter is fitted to the measured outcome. The theoretical results (Lemperts 1–5 and the worst-case search algorithms in the appendices) establish that WorstCR(F,Fhat) can be computed efficiently for assortment, pricing, and newsvendor, given F and Fhat, and do not presuppose the empirical conclusion. The only documented risk is possible training-data contamination of the LLM (few-shot examples drawn from the same population as F, and no contamination check), which concerns external validity or data leakage, not circular reasoning. There is no self-definitional step, no fit-then-predict reduction, no self-citation chain used as a load-bearing justification, and no renamed known result presented as a derivation. The paper is self-contained in that all empirical comparisons are made against external ground-truth datasets, so the central evaluation does not reduce to its own inputs.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No fitted free parameters appear in the central derivations. The main assumptions are domain-level: the choice of dataset subsets as ground truth, representativeness of the average-case parameter distributions, and no LLM training-data contamination. The latter is the weakest and is not verified by the paper.

assumptions (6)

domain assumption Ground-truth F is the empirical distribution over a fixed subset of each public dataset (first 600 sushi respondents; first 100 cocoa WTP respondents; 30-week H&M trouser demand).
All competitive-ratio evaluations compare against this F; if the dataset subset is not the true population distribution, the measured decision quality is not the real-world decision quality.
domain assumption The LLM has not memorized the evaluation datasets; its outputs reflect general world knowledge, not test-data leakage.
No contamination check is reported; few-shot examples are drawn from the same survey population as F, so leakage could inflate LLM-SAA performance.
ad hoc to paper The average-case parameter distributions (Unit/Random/Hard sizes for assortment, cost ranges Unif[0,32]/[0,66]/[0,100] for pricing, q~Unif[0.01,0.99] for newsvendor) are representative of relevant real instances.
These ranges are chosen by the authors; worst-case/average-case concordance is demonstrated only for these choices.
standard math For the assortment problem, rewards are monotone in set inclusion, so an optimal assortment can be taken to be inclusion-maximal; the LP feasibility conditions in Section A characterize when a given pair (a*, â) is realizable.
Used in Lemma 1 and the WorstCR algorithm to reduce the search to maximal feasible sets and to exclude bad assortments.
standard math For pricing with discrete support, the optimal price lies in the support (or infinity); for newsvendor with discrete uniform demand, the optimal inventory is a demand quantile.
Used in Lemmas 2 and 4 to restrict the search for optimal actions under F and Fhat.
standard math For fixed a* and â, the competitive-ratio objective is monotone in the parameter (cost c or quantile q), so the worst case occurs at a breakpoint.
Used in Lemmas 3 and 5 in the pricing and newsvendor worst-case algorithms.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM-SAA: LLM-persona Generated Distributions for Decision-making." pith.science (2026). https://pith.science/paper/3UZXJHOS

@misc{pith2026260206357,
  author       = {Pith},
  title        = {Pith review of: LLM-SAA: LLM-persona Generated Distributions for Decision-making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3UZXJHOS}},
  note         = {Machine review of arXiv:2602.06357}
}

read the original abstract

LLMs can generate a wealth of data, ranging from simulated personas imitating human valuations and preferences, to demand forecasts based on world knowledge. But how well do such LLM-generated distributions support downstream decision-making? For example, when pricing a new product, a firm could prompt an LLM to simulate how much consumers are willing to pay based on a product description, but how useful is the resulting distribution for optimizing the price? We refer to this approach as LLM-SAA, in which an LLM is used to construct an estimated distribution and the decision is then optimized under that distribution. In this paper, we study metrics to evaluate the quality of these LLM-generated distributions, based on the decisions they induce. Taking three canonical decision-making problems (assortment optimization, pricing, and newsvendor) as examples, we find that LLM-generated distributions are practically useful, especially in low-data regimes. We also show that decision-agnostic metrics such as Wasserstein distance can be misleading when evaluating these distributions for decision-making.

Figures

Figures reproduced from arXiv: 2602.06357 by the authors.

**Figure 2.** Pricing results, displaying means across 20 generations and 6 ground truths, with 95% [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

**Figure 3.** Newsvendor results across four LLM models, displaying means across 300 items with [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Illustration of the enumeration and pruning procedure, in our special case where [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Survival functions of the willingness-to-pay distributions for the ground truth distribu [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 7 linked inside Pith

[3]

Y. Duan, Y. Hu, and J. Jiang. Ask, clarify, optimize: Human-llm agent collaboration for smarter inventory control.arXiv preprint arXiv:2601.00121,

arXiv
[6]

Accessed: 2026-01-31

URLhttps://hbr.org/2025/12/ when-supply-chains-become-autonomous. Accessed: 2026-01-31. B. S. Manning, K. Zhu, and J. J. Horton. Automated social science: Language models as scientist and subjects. Working Paper 32381, National Bureau of Economic Research,

2025
[11]

doi: 10.1287/mnsc.2020.3657. K. T. Talluri and G. J. van Ryzin.The Theory and Practice of Revenue Management. Springer,

arXiv 2020
[12]

H. Wang, G. Chen, K. Talluri, and X. Li. Omgpt: A sequence modeling framework for data-driven operational decision making.arXiv preprint arXiv:2505.13580,

arXiv
[13]

M. Wang, D. J. Zhang, and H. Zhang. Large language models for market research: A data- augmentation approach.arXiv preprint arXiv:2412.19363,

arXiv
[14]

Q. Xie, Q. Feng, T. Zhang, Q. Li, L. Yang, Y. Zhang, R. Feng, L. He, S. Gao, and Y. Zhang. Human simulacra: Benchmarking the personification of large language models.arXiv preprint arXiv:2402.18180,

arXiv
[15]

C. Zhou, J. Yang, L. Xin, Y. Chen, Z. He, and D. Ge. Auto-formulating dynamic programming problems with large language models.arXiv preprint arXiv:2507.11737,

arXiv
[16]

upgraded

26 •Persona-sampling: We generate a pool of 100 willingness-to-pay values for each products, prompting the LLM with a different persona each time. We subsample 50 willingness-to-pay values to form an estimated distribution bF, and repeat to form 20 estimated distributions in this way. •Batch-generation: We generate 25 willingness-to-pay values for each pr...

2022

Show all 16 references

[1975]

doi: 10.2307/2346567. Y. Qu and J. Wang. Performance and biases of large language models in public opinion sim- ulation.Humanities and Social Sciences Communications, 11:1095,

doi
[2013]

doi: 10.1287/mnsc.1120.1625. Y. Gao, D. Lee, G. Burtch, and S. Fazelpour. Take caution in using llms as human surrogates. Proceedings of the National Academy of Sciences, 122(24):e2501660122,
[2021]

Samuel, H

V. Samuel, H. P. Zou, Y. Zhou, S. Chaudhari, A. Kalyan, T. Rajpurohit, A. Deshpande, K. Narasimhan, and V. Murahari. Personagym: Evaluating persona agents and llms.arXiv preprint arXiv:2407.18416,

arXiv
[2022]

Kaggle. J. Liu, Z. Chen, and Y. Zhong. Large language newsvendor: Decision biases and cognitive mecha- nisms.arXiv preprint arXiv:2512.12552,
[2023]

Cao and M

S. Cao and M. Hu. Llm collusion.arXiv preprint arXiv:2601.01279,

arXiv
[2024]

Meister, C

N. Meister, C. Guestrin, and T. B. Hashimoto. Benchmarking distributional alignment of large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Lo...

2025
[2025]

URLhttps://arxiv.org/abs/2502.15840. J. F. Ballesteros, J. J. Schouteten, A. Otilla, R. I. Ramirez, X. Gellynck, J. Casaul, and H. De Steur. Does award and origin labeling influence consumers’ willingness-to-pay beyond sensory cues? an experimental auction on improved philippi...

arXiv
[2026]

16 J. Suh, E. Jahanparast, S. Moon, M. Kang, and S. Chang. Language model fine-tuning on scaled survey data for predicting distributions of public opinions.arXiv preprint arXiv:2502.16761,

arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [3]

Y. Duan, Y. Hu, and J. Jiang. Ask, clarify, optimize: Human-llm agent collaboration for smarter inventory control.arXiv preprint arXiv:2601.00121,

arXiv

[2] [6]

Accessed: 2026-01-31

URLhttps://hbr.org/2025/12/ when-supply-chains-become-autonomous. Accessed: 2026-01-31. B. S. Manning, K. Zhu, and J. J. Horton. Automated social science: Language models as scientist and subjects. Working Paper 32381, National Bureau of Economic Research,

2025

[3] [11]

doi: 10.1287/mnsc.2020.3657. K. T. Talluri and G. J. van Ryzin.The Theory and Practice of Revenue Management. Springer,

arXiv 2020

[4] [12]

H. Wang, G. Chen, K. Talluri, and X. Li. Omgpt: A sequence modeling framework for data-driven operational decision making.arXiv preprint arXiv:2505.13580,

arXiv

[5] [13]

M. Wang, D. J. Zhang, and H. Zhang. Large language models for market research: A data- augmentation approach.arXiv preprint arXiv:2412.19363,

arXiv

[6] [14]

Q. Xie, Q. Feng, T. Zhang, Q. Li, L. Yang, Y. Zhang, R. Feng, L. He, S. Gao, and Y. Zhang. Human simulacra: Benchmarking the personification of large language models.arXiv preprint arXiv:2402.18180,

arXiv

[7] [15]

C. Zhou, J. Yang, L. Xin, Y. Chen, Z. He, and D. Ge. Auto-formulating dynamic programming problems with large language models.arXiv preprint arXiv:2507.11737,

arXiv

[8] [16]

upgraded

26 •Persona-sampling: We generate a pool of 100 willingness-to-pay values for each products, prompting the LLM with a different persona each time. We subsample 50 willingness-to-pay values to form an estimated distribution bF, and repeat to form 20 estimated distributions in this way. •Batch-generation: We generate 25 willingness-to-pay values for each pr...

2022