Pith. sign in

REVIEW 3 major objections 5 minor 146 references

Large language models do not satisfy the law of total probability, yet their subgroup-level estimates often beat direct population-level estimates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 23:38 UTC pith:BJUFUST2

load-bearing objection The macro fallacy finding is real and worth engaging; the order-consistency certificate rests on an untested context-independence assumption that needs fixing before the paper's stronger claims stand. the 3 major comments →

arxiv 2607.15277 v1 pith:BJUFUST2 submitted 2026-07-16 cs.CL

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

Patrik Wolf , Thomas Kleine Buening , Andreas Krause , Celestine Mendler-D\"unner This is my paper
classification cs.CL
keywords statistical self-consistencylaw of total probabilityin-context learningpersona promptingmacro fallacybinary conditioning treesdistributional alignmentreference-free evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests whether in-context learning behaves like conditional inference by checking the law of total probability: a population-level estimate should equal the prior-weighted sum of estimates over any valid partition. Using binary trees that recursively split a population into finer subgroups, the authors prompt models for each subgroup and recombine the answers. They find widespread violations: direct aggregate estimates systematically disagree with reconstructed aggregates. The reconstruction is often better aligned with human reference data, a pattern they call the macro fallacy. The paper concludes that statistical self-consistency is a currently unsaturated, reference-free evaluation criterion.

Core claim

The central discovery is that LLM estimates are not statistically self-consistent: for the same target quantity, a model's direct marginal estimate and its prior-weighted aggregate over fine-grained conditional estimates diverge, and the divergence is systematic. Direct population-level prompting tends to underuse subgroup knowledge; reconstructed estimates from deeper tree levels are often closer to human reference data. This macro fallacy persists across models, income thresholds, and tree structures, and can be partially recovered by implicit 'micro-to-macro' prompting within a single query. The authors introduce two reference-free checks—split consistency (does an estimate equal the weig

What carries the argument

The central object is the binary conditioning tree: a population is recursively split by binary attributes, with each level forming a valid partition. Each node's verbalized subpopulation description is prompted to elicit an LLM estimate; the law of total probability supplies the aggregation identity that links node estimates to parent and root estimates. Split consistency and order consistency are the local checks derived from this identity, with a multi-step aggregation bound and a pairwise-order certification. The law of total variance is used to explain why refinement reduces residual uncertainty.

Load-bearing premise

The order-consistency score assumes that the effect of swapping two constraints in a prompt depends only on the pair of constraints, not on the surrounding prompt context; if that assumption fails, pairwise order checks no longer certify consistency of longer conditioning events.

What would settle it

Take a conditioning event with two constraints, embed the same pair in several different prompt contexts, and measure the change in the model's estimate when the two constraints are swapped. If the swap-induced discrepancy varies across contexts, the context-independent pairwise order discrepancy assumption fails, and the order-consistency scores built on pairwise checks would not measure what they claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Population-level alignment cannot be read off subpopulation accuracy: two statistically equivalent ways of asking the same question can yield incompatible model estimates.
  • Self-consistency can be evaluated without ground-truth labels, so it applies to forecasting and synthetic settings where reference data does not exist.
  • Explicitly decomposing a query into subgroup estimates and recombining them is a practical, sometimes single-prompt intervention that improves alignment.
  • Because consistency constraints are directionless, model developers can choose to anchor them at the better-aligned fine-grained estimates, pushing aggregate estimates upward.
  • General capability improvements do not automatically increase statistical self-consistency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the macro fallacy suggests a form of averaging bias—models compress heterogeneous subpopulations toward a central tendency, so forcing decomposition exposes variance that direct prompts wash out.
  • Editorial inference: consistency scores could be used as a training signal or reward term, but only if the direction of the anchor is chosen carefully; otherwise optimizing consistency alone could pull fine-grained estimates toward a biased aggregate.
  • Editorial inference: the same partitioning scaffold could be applied to non-demographic contexts, such as decomposing a prediction by evidence scenarios, to build reference-free forecast calibration checks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework for evaluating whether LLM estimates behave as conditional distributions under the law of total probability. Using binary conditioning trees over populations or scenario spaces, the authors elicit estimates for subpopulations at varying granularity, aggregate them with estimated priors, and compare the resulting reconstructions against direct root-level estimates and against human-survey ground truth. They report two reference-free consistency diagnostics: split consistency, checking that a direct estimate agrees with the prior-weighted aggregate of its two refinements, and order consistency, checking invariance to the order of conditioning constraints. The paper documents widespread violations of these checks across several frontier model families and domains (ACS income and commute time, WVS opinions, synthetic forecasting), and identifies the 'macro fallacy': reconstructed aggregates from finer-grained subpopulation estimates are often better aligned with ground-truth survey data than direct aggregate prompting. They also propose micro-to-macro prompting as a lightweight partial remedy and argue that statistical self-consistency is an unsaturated, reference-free evaluation axis.

Significance. If the claims hold, the paper makes a useful methodological contribution: it turns a basic probabilistic identity into a structured, reference-free evaluation protocol and demonstrates that current LLMs fail it in a systematic direction. The breadth of models and domains, the public code link, and the clean proofs of Propositions 2 and 7 are strengths. The macro-fallacy result, if robust, has practical implications for persona-based simulation and forecasting. However, the significance is partly conditional on the order-consistency certificate, which rests on an unverified behavioral assumption, and on the statistical support for the consistency-score comparisons, which are presented without uncertainty quantification.

major comments (3)
  1. [Section 5.2, Assumption 1 and Proposition 6] Proposition 6 claims that pairwise order-consistency checks control order consistency for arbitrary conditioning events. The proof in Appendix C.2 invokes Assumption 1 at the step where each adjacent transposition in a longer ordering is assumed to induce the same discrepancy as in the isolated two-attribute event. The paper calls this 'a natural approximation' but provides no empirical or theoretical support. This is load-bearing because order consistency is one of only two self-consistency axes, and the layer-permutation guarantee in Proposition 7 also builds on it. The cited works (Chen et al. 2024, Guan et al. 2025) suggest that LLM order sensitivity is context-dependent, which is direct contrary evidence. Please provide a concrete test of Assumption 1—e.g., compare swap discrepancies with and without additional prefix/suffix constraints on the same attribute set—or weaken the claims
  2. [Section 6, Tables 1-3 and Figure 6] All split and order consistency scores are reported as point estimates without confidence intervals or repeated-sample variability. This is especially consequential because the check sets are small: for d=2 attributes, CSC has 6 checks and COC has 4 checks, so a single check changes the score by 1/6 or 1/4. The WVS tables are similarly coarse. The paper's claims that scores are 'widespread violations', that scores 'differ substantially' across tasks, and that capability does not systematically improve self-consistency (Figure 6) are therefore not statistically supported at the reported precision. Please add bootstrap confidence intervals over LLM samples, or report per-check results and the number of independent elicitations used for each score.
  3. [Appendix E.9 / Section 6.1] In the WVS experiments, the income split is defined only as 'low' versus 'high' and the interpretation is explicitly left to the model. Split consistency presupposes that the two refinements form a valid partition of the target population. If the model's internal threshold is not complementary—e.g., if 'low' and 'high' leave a middle region or overlap—the law-of-total-probability check is not well-defined for that attribute. Because WVS is one of the three domains used to support the cross-domain claim, please either fix a concrete income threshold, validate complementarity (e.g., by eliciting both branch definitions and checking that their union is exhaustive), or explicitly restrict the WVS split-consistency claims to the age split.
minor comments (5)
  1. [Section 3.4] The notation overloads X: it is both the base population set and the random variable representing an individual. Clarify, e.g., by writing the population as X and the random vector as X~.
  2. [Section 5.2] Definition 4 defines order consistency over all |J|! orderings, but the order-consistency score in Definition 5 only evaluates two-attribute events. Please state clearly that the score is a pairwise proxy whose extension to arbitrary events relies on Assumption 1, and that without Assumption 1 the score only measures pairwise order invariance.
  3. [Section 8 / Figure 6] The phrase 'we find no evidence that our consistency scores correlate with standard benchmark metrics' is stronger than the evidence presented. Figure 6 shows no monotone within-family trend against the AAI index, but no correlation coefficient or statistical test is reported. Please either soften the claim or add a quantitative analysis.
  4. [Section 5.1] The tolerance epsilon=0.02 is used throughout, but there is no sensitivity analysis. Since normalized Wasserstein distances and absolute differences are on different scales for binary vs. ordered-categorical outcomes, a short robustness table across epsilon values would strengthen the benchmark interpretation.
  5. [Appendix E.9] The WVS income split categories 'low' and 'high' appear in the prompt template without a precise definition, while the age split is explicit. Consider adding the exact prompt wording for the income split to the appendix so readers can assess the validity of the partition.

Circularity Check

0 steps flagged

No material circularity: the consistency checks are direct instantiations of the law of total probability and permutation invariance, and the macro-fallacy claim is benchmarked against external ACS/WVS ground truth; Assumption 1 is a validity caveat, not a circular derivation.

full rationale

The paper's derivation chain is not circular. Split consistency (Definition 1, Eq. 5) directly instantiates the law of total probability by comparing a model's direct estimate for a conditioning event with the prior-weighted aggregate of its estimates for a partition; this is an intentional internal diagnostic, not a fitted parameter renamed as a prediction. Order consistency (Definitions 4-5) checks invariance of estimates under constraint permutations, again a direct identity check. Proposition 6 and Proposition 7 are conditional theorems with explicit proofs relying on Assumption 1 (context-independent pairwise order discrepancy); if that assumption fails, the certificate is unsupported, but the argument does not reduce to its inputs by construction—it is a stated modeling assumption. The macro fallacy is an empirical comparison of reconstructed aggregates against external ACS ground truth (Eq. 4), not a quantity implied by the fitting procedure. Priors and conditionals are elicited independently from the LLM and aggregated via the exact identity in Eq. (1); no parameter is fitted to the target values and then reported as a prediction. The paper explicitly acknowledges that self-consistency is only a necessary condition and that a model can be consistent yet misaligned. Background citations to prior work by the authors (e.g., Cruz et al. 2024, Dominguez-Olmedo et al. 2024) are used for context or implementation tooling, not as load-bearing evidence for the central claims. The main robustness concern—Assumption 1's context independence—is a validity/assumption risk, not circularity within the derivation.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The central claim does not introduce new physical or theoretical entities. The main hand-chosen inputs are evaluation thresholds and data-driven split choices, which are disclosed but affect the numerical results. Assumption 1 is the most delicate modeling assumption and is load-bearing for the order-consistency sufficiency claim.

free parameters (3)
  • consistency tolerance epsilon = 0.02
    Chosen by hand for all split and order consistency scores; the 'widespread violations' claim depends on this threshold and no sensitivity analysis over epsilon is reported.
  • ACS split criteria (age and class of worker) = age 31-68 vs rest; COW values 1-7 vs 8,9,n/a
    Selected from ACS survey data to separate lower- and higher-income branches while keeping priors balanced. This data-driven split selection can inflate the macro-fallacy gain because it is tuned to the target variable.
  • income bin edges = [-11.5k, 1k, 25k, 60k, 1849k] USD
    Bin edges are chosen so that the root income distribution is approximately uniform; alignment errors and the magnitude of the macro fallacy depend on this binning.
axioms (6)
  • standard math The law of total probability holds for valid partitions of the population (Equation 2).
    This is the external benchmark identity the paper uses to define consistency; it is standard probability theory applied to LLM-estimated quantities.
  • standard math The discrepancy metric is jointly convex under mixtures.
    Needed for the triangle-inequality argument in Proposition 2; the paper's Wasserstein-1 and TV metrics satisfy this, but it is an explicit technical requirement.
  • ad hoc to paper Assumption 1: the effect of swapping two adjacent constraints is independent of surrounding prompt context.
    Explicitly introduced in Section 5.2 and load-bearing for Proposition 6, which claims pairwise order checks certify order consistency for arbitrary conditioning events.
  • domain assumption Verbalized subpopulation descriptions elicit valid conditional probability estimates from LLMs.
    The entire measurement pipeline treats natural-language persona descriptions as prompts that induce conditional estimates; if this fails, observed violations could be elicitation artifacts rather than genuine inconsistency.
  • domain assumption The binary splits for WVS 'low' vs 'high' income are interpreted by the model as complementary and exhaustively partitioning the population.
    Appendix E.9 says the interpretation of low/high is left to the model; the law of total probability requires the two options to form a valid partition in the model's interpretation.
  • domain assumption The ACS 2024 survey-weighted data accurately represents the U.S. population ground truth.
    All alignment and macro-fallacy measurements for the U.S. compare LLM estimates against ACS; if ACS is not the appropriate reference for the prompted population, alignment claims change.

pith-pipeline@v1.3.0-alltime-deepseek · 37100 in / 11792 out tokens · 124756 ms · 2026-08-01T23:38:01.534432+00:00 · methodology

0 comments
read the original abstract

In-context learning is commonly interpreted as a form of conditional inference, in which the prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Applying this protocol across problem domains and state-of-the-art frontier models, we show widespread violations of basic consistency properties. An in-depth study of persona prompting reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. This effect persists across variations in tree structure and estimation task, and can be partially recovered through implicit prompting. Together, these findings suggest that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

Figures

Figures reproduced from arXiv: 2607.15277 by Andreas Krause, Celestine Mendler-D\"unner, Patrik Wolf, Thomas Kleine Buening.

Figure 1
Figure 1. Figure 1: Reconstructing aggregate estimates from subpopulation estimates. A binary conditioning tree recursively partitions a base population into increasingly specific subpopulations. For each subpopulation, we elicit an LLM estimate via in-context learning by providing a verbalized description of the split criteria as context. The pink and violet branches provide examples of the conditioning attribute introduced … view at source ↗
Figure 2
Figure 2. Figure 2: Ground truth and LLM-elicited income distributions across partitions of the population. We construct a binary conditioning tree from the 2024 ACS population by recursively slicing the population with sociodemographic attributes. The first split criterion corresponds to age (AGEP) and the second to class of worker (COW). Each node reports the distribution of yearly income over four bins. For each bin, the s… view at source ↗
Figure 3
Figure 3. Figure 3: Relative gain of reconstructed aggregates over direct prompting. We report the relative alignment gain of the reconstructed estimate, defined as 1 − AErr(ℓ)/AErr(0), as a function of tree depth ℓ, where AErr(0) denotes the alignment error of direct aggregate prompting. Positive values indicate that reconstructing the population-level estimate from subgroup estimates improves alignment relative to direct pr… view at source ↗
Figure 4
Figure 4. Figure 4: Disentangling the sources of estimation error. The left and middle panels separate the subgroup conditional estimates and subgroup priors in the reconstructed aggregate estimate. Node-wise conditional errors tend to improve over the root node, yet they vary substantially across the tree. LLM-estimated priors become less aligned with survey-based priors at deeper levels. The right panel illustrates the vari… view at source ↗
Figure 5
Figure 5. Figure 5: Benefit of implicit micro-to-macro prompting on ACS income estimation. We compare direct aggregate prompting with micro-to-macro prompting, which asks the model to first reason over relevant subpopulations before returning an aggregate prediction, while leaving the subpopulations implicit. Unlike explicit tree-based aggregation, the decomposition is not fixed in advance but chosen implicitly by the model. … view at source ↗
Figure 6
Figure 6. Figure 6: Model comparison. We report the split consistency score SCε(A) and order consistency score OCε(A) at tolerance ε = 0.02 for the thresholded ACS income prediction task with τ = 40k USD. The attribute set A consists of age and employment status as used in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Conditional aggregation gain using ground-truth priors. Reconstructing population￾level estimates with ground-truth subgroup priors from increasingly fine-grained subgroup estimates consistently improves alignment relative to direct aggregate prompting. Error bars show 90 % confidence intervals computed by bootstrapping with 1000 samples. prompting. We observe that reconstructed estimates from deeper parti… view at source ↗
Figure 8
Figure 8. Figure 8: Reconstructed aggregate income distributions with ground-truth priors. We reconstruct the population-level income distribution at each tree level by aggregating LLM-estimated subgroup distributions using ACS subgroup priors. Solid bars show the survey-based aggregate distribution, while hatched bars show the aggregate reconstructed from LLM-estimated subgroup conditionals. The colored border indicates the … view at source ↗
Figure 9
Figure 9. Figure 9: LLM-estimated priors become less aligned at deeper tree levels. For each tree level, we compare the LLM-estimated distribution over subgroups with the survey-based ground-truth prior distribution from the ACS data using total variation (TV) distance. Across models, prior alignment deteriorates as the tree becomes more fine-grained. Error bars show 90 % confidence intervals computed by bootstrapping with 10… view at source ↗
Figure 10
Figure 10. Figure 10: Tree interpretation of split and order consistency. Within each tree, a parent estimate should agree with the prior-weighted aggregate of its two child estimates. Interchanging two adjacent layers preserves the leaf events but changes both the order in which their constraints are presented and the intermediate conditioning events exposed by the tree. Split consistency. Consider a node representing a condi… view at source ↗
Figure 11
Figure 11. Figure 11: Micro-to-macro prompting on GlobalOpinionQA. We compare direct aggregate prompting with micro-to-macro prompting on selected binary-response questions from GlobalOpinionQA. For each question–country pair, the model predicts the probability that a randomly sampled person from the specified country would choose the first of the two binary answer options. We evaluate absolute error with respect to the survey… view at source ↗
Figure 12
Figure 12. Figure 12: Prompt sensitivity of probability elicitation. Results are shown for GPT-5.4 without reasoning, using ground-truth priors and 100 repeated LLM samples. Direct estimates are obtained from population-level prompts, whereas reconstructed estimates are obtained by aggregating subgroup-level estimates via the law of total probability using ground-truth priors. The population is decomposed according to employme… view at source ↗
Figure 13
Figure 13. Figure 13: Mean income across attribute-induced subgroups. We partition the ACS population by all attainable values of a single attribute and report the survey-weighted mean yearly income in USD for each resulting subgroup. For the age attribute in Figure 13a, subgroup values are displayed in their natural order, and the binary split is defined by thresholding the corresponding subgroup mean incomes (dashed line). F… view at source ↗
Figure 14
Figure 14. Figure 14: Aggregation gain across models and tree levels. We report the aggregation gain 1 − AErr(ℓ)/AErr(0) with AErr(ℓ) defined in (4). Positive values indicate that reconstructing the population-level estimate from subgroup estimates improves alignment relative to direct prompting, while negative values indicate worse alignment. Each curve corresponds to a different model. Error bars show 90 % confidence interva… view at source ↗
Figure 15
Figure 15. Figure 15: reports the complete win-matrix analysis for micro-to-macro prompting on GlobalOpinionQA across all evaluated models. Germany Japan France U.S. Australia Italy Spain 55 136 573 839 1108 1445 1512 Question GPT-5.4 Win rate = 63.3 % Germany Japan France U.S. Australia Italy Spain 55 136 573 839 1108 1445 1512 3.6 Plus Win rate = 83.7 % Germany Japan France U.S. Australia Italy Spain 55 136 573 839 1108 1445… view at source ↗
Figure 16
Figure 16. Figure 16: visualizes the population-level ACS income distributions reconstructed from each tree layer using LLM-estimated subgroup priors. Income 0.0 0.2 P Level 0 Income 0.0 0.2 P Level 1 Income 0.0 0.2 P Level 2 ACS LLM [−11.5k, 1) USD [1, 25k) USD [25k, 60k) USD [60k, 1849k] USD [PITH_FULL_IMAGE:figures/full_fig_p042_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

146 extracted references · 2 canonical work pages

  1. [1]

    Cultivating Pluralism In Algorithmic Monoculture:

    Lily H Zhang and Smitha Milli and Karen Long Jusko and Jonathan Smith and Brandon Amos and Wassim Bouaziz and Manon Revel and Jack Kussman and Yasha Sheynin and Lisa Titus and Bhaktipriya Radharapu and Jane Yu and Vidya Sarma and Kristopher Rose and Maximilian Nickel , year = 2026, booktitle =. Cultivating Pluralism In Algorithmic Monoculture:

    2026

  2. [2]

    In-context learning is provably

    Wakayama, Tomoya and Suzuki, Taiji , year = 2025, journal =. In-context learning is provably

    2025

  3. [3]

    Incoherent probability judgments in large language models , author =

  4. [4]

    An Information-Theoretic Analysis of In-Context Learning , author =

  5. [5]

    Learning to summarize user information for personalized reinforcement learning from human feedback , author =

  6. [6]

    Advances in Neural Information Processing Systems , volume = 36, pages =

    The learnability of in-context learning , author =. Advances in Neural Information Processing Systems , volume = 36, pages =

  7. [7]

    Advances in Neural Information Processing Systems , volume = 37, pages =

    To believe or not to believe your llm: Iterative prompting for estimating epistemic uncertainty , author =. Advances in Neural Information Processing Systems , volume = 37, pages =

  8. [8]

    Zhehao Zhang and Ryan A. Rossi and Branislav Kveton and Yijia Shao and Diyi Yang and Hamed Zamani and Franck Dernoncourt and Joe Barrow and Tong Yu and Sungchul Kim and Ruiyi Zhang and Jiuxiang Gu and Tyler Derr and Hongjie Chen and Junda Wu and Xiang Chen and Zichao Wang and Subrata Mitra and Nedim Lipka and Nesreen K. Ahmed and Yu Wang , year = 2025, jo...

    2025

  9. [10]

    Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models , author =

  10. [11]

    2510.15674 , archiveprefix =

    CarBoN: Calibrated Best-of-N Sampling Improves Test-time Reasoning , author =. 2510.15674 , archiveprefix =

    arXiv

  11. [12]

    2510.00931 , archiveprefix =

    Making, not Taking, the Best of N , author =. 2510.00931 , archiveprefix =

    arXiv

  12. [13]

    2510.13902 , archiveprefix =

    Investigating Political and Demographic Associations in Large Language Models Through Moral Foundations Theory , author =. 2510.13902 , archiveprefix =

    arXiv

  13. [14]

    2509.04784 , archiveprefix =

    Post-training Large Language Models for Diverse High-Quality Responses , author =. 2509.04784 , archiveprefix =

    arXiv

  14. [15]

    1904.09751 , archiveprefix =

    The Curious Case of Neural Text Degeneration , author =. 1904.09751 , archiveprefix =

    Pith/arXiv arXiv 1904

  15. [16]

    2407.01082 , archiveprefix =

    Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs , author =. 2407.01082 , archiveprefix =

    arXiv

  16. [17]

    2002.00632 , archiveprefix =

    Effective Diversity in Population Based Reinforcement Learning , author =. 2002.00632 , archiveprefix =

    Pith/arXiv arXiv 2002

  17. [18]

    2502.14037 , archiveprefix =

    DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generation , author =. 2502.14037 , archiveprefix =

    arXiv

  18. [19]

    2202.06417 , archiveprefix =

    A Contrastive Framework for Neural Text Generation , author =. 2202.06417 , archiveprefix =

    Pith/arXiv arXiv

  19. [20]

    2210.14140 , archiveprefix =

    Contrastive Search Is What You Need For Neural Text Generation , author =. 2210.14140 , archiveprefix =

    Pith/arXiv arXiv

  20. [21]

    Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using

    Tevin Atwal and Chan Nam Tieu and Yefeng Yuan and Zhan Shi and Yuhong Liu and Liang Cheng , year = 2025, url =. Privacy-Preserving Synthetic Review Generation with Diverse Writing Styles Using. 2507.18055 , archiveprefix =

    Pith/arXiv arXiv 2025

  21. [22]

    Personas with Attitudes: Controlling

    Leon Fröhling and Gianluca Demartini and Dennis Assenmacher , year = 2024, url =. Personas with Attitudes: Controlling. 2410.11745 , archiveprefix =

    Pith/arXiv arXiv 2024

  22. [23]

    Multilingual Prompting for Improving

    Qihan Wang and Shidong Pan and Tal Linzen and Emily Black , year = 2025, url =. Multilingual Prompting for Improving. 2505.15229 , archiveprefix =

    arXiv 2025

  23. [24]

    Exploring Model Label Variation , author =

    Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation , author =. 2307.12973 , archiveprefix =

    Pith/arXiv arXiv

  24. [25]

    2402.19379 , archiveprefix =

    Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy , author =. 2402.19379 , archiveprefix =

    Pith/arXiv arXiv

  25. [27]

    and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer M

    Bisbee, James and Clinton, Joshua D. and Dorff, Cassy and Kenkel, Brenton and Larson, Jennifer M. , year = 2024, journal =. Synthetic Replacements for Human Survey Data?

    2024

  26. [28]

    Proceedings of the 40th International Conference on Machine Learning , publisher =

    Whose Opinions Do Language Models Reflect? , author =. Proceedings of the 40th International Conference on Machine Learning , publisher =

  27. [30]

    2306.06826 , archiveprefix =

    When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset , author =. 2306.06826 , archiveprefix =

    Pith/arXiv arXiv

  28. [31]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , publisher =

    The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , publisher =. doi:10.18653/v1/2023.acl-short.88 , url =

    doi:10.18653/v1/2023.acl-short.88 2023

  29. [33]

    Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume = 8, number = 1, pages =

    Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency , author =. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume = 8, number = 1, pages =. doi:10.1609/aies.v8i1.36553 , url =

    doi:10.1609/aies.v8i1.36553

  30. [34]

    Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with

    Huaman Sun and Jiaxin Pei and Minje Choi and David Jurgens , year = 2025, url =. Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with. 2311.09730 , archiveprefix =

    Pith/arXiv arXiv 2025

  31. [35]

    2504.02234 , archiveprefix =

    LLM Social Simulations Are a Promising Research Method , author =. 2504.02234 , archiveprefix =

    Pith/arXiv arXiv

  32. [37]

    2510.11686 , archiveprefix =

    Representation-Based Exploration for Language Models: From Test-Time to Post-Training , author =. 2510.11686 , archiveprefix =

    Pith/arXiv arXiv

  33. [38]

    Does Reinforcement Learning Really Incentivize Reasoning Capacity in

    Yang Yue and Zhiqi Chen and Rui Lu and Andrew Zhao and Zhaokai Wang and Yang Yue and Shiji Song and Gao Huang , year = 2025, url =. Does Reinforcement Learning Really Incentivize Reasoning Capacity in. 2504.13837 , archiveprefix =

    Pith/arXiv arXiv 2025

  34. [39]

    1611.04717 , archiveprefix =

    \#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning , author =. 1611.04717 , archiveprefix =

    Pith/arXiv arXiv

  35. [40]

    1705.05363 , archiveprefix =

    Curiosity-driven Exploration by Self-supervised Prediction , author =. 1705.05363 , archiveprefix =

    Pith/arXiv arXiv

  36. [41]

    1703.07608 , archiveprefix =

    Deep Exploration via Randomized Value Functions , author =. 1703.07608 , archiveprefix =

    Pith/arXiv arXiv

  37. [42]

    Monoculture or Multiplicity: Which Is It? , author =

  38. [43]

    2502.04262 , archiveprefix =

    Efficient Randomized Experiments Using Foundation Models , author =. 2502.04262 , archiveprefix =

    arXiv

  39. [44]

    Sociological Methods & Research , volume = 54, number = 3, pages =

    The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations , author =. Sociological Methods & Research , volume = 54, number = 3, pages =. doi:10.1177/00491241251326865 , url =. https://doi.org/10.1177/00491241251326865 , abstract =

    doi:10.1177/00491241251326865

  40. [45]

    2306.04746 , archiveprefix =

    Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models , author =. 2306.04746 , archiveprefix =

    Pith/arXiv arXiv

  41. [46]

    Proceedings of the National Academy of Sciences , volume = 121, number = 21, pages =

    Can Generative AI improve social science? , author =. Proceedings of the National Academy of Sciences , volume = 121, number = 21, pages =. doi:10.1073/pnas.2314021121 , url =. https://www.pnas.org/doi/pdf/10.1073/pnas.2314021121 , abstract =

    doi:10.1073/pnas.2314021121

  42. [47]

    Alleviating shifted distribution in human preference alignment through meta-learning , author =. Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence , publisher =. doi:10.1609...

    work page doi:10.1609/aaai.v39i22.34552

  43. [48]

    2601.05184 , archiveprefix =

    Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loop , author =. 2601.05184 , archiveprefix =

    arXiv

  44. [49]

    2512.23461 , archiveprefix =

    Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance , author =. 2512.23461 , archiveprefix =

    Pith/arXiv arXiv

  45. [50]

    2305.14929 , archiveprefix =

    Aligning Language Models to User Opinions , author =. 2305.14929 , archiveprefix =

    Pith/arXiv arXiv

  46. [51]

    Advances in Neural Information Processing Systems , volume = 37, pages =

    Questioning the survey responses of large language models , author =. Advances in Neural Information Processing Systems , volume = 37, pages =

  47. [52]

    2404.13076 , archiveprefix =

    LLM Evaluators Recognize and Favor Their Own Generations , author =. 2404.13076 , archiveprefix =

    Pith/arXiv arXiv

  48. [53]

    2305.01937 , archiveprefix =

    Can Large Language Models Be an Alternative to Human Evaluations? , author =. 2305.01937 , archiveprefix =

    Pith/arXiv arXiv

  49. [54]

    2306.05685 , archiveprefix =

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author =. 2306.05685 , archiveprefix =

    Pith/arXiv arXiv

  50. [55]

    Assessing Diversity Collapse in Reasoning , author =

  51. [56]

    Nature , volume = 631, number = 8022, pages =

    AI models collapse when trained on recursively generated data , author =. Nature , volume = 631, number = 8022, pages =. doi:10.1038/s41586-024-07566-y , isbn =

    doi:10.1038/s41586-024-07566-y

  52. [57]

    2601.20802 , archiveprefix =

    Reinforcement Learning via Self-Distillation , author =. 2601.20802 , archiveprefix =

    Pith/arXiv arXiv

  53. [58]

    What are the odds?

    Paruchuri, Akshay and Garrison, Jake and Liao, Shun and Hernandez, John B and Sunshine, Jacob and Althoff, Tim and Liu, Xin and McDuff, Daniel , year = 2024, booktitle =. What are the odds?

    2024

  54. [59]

    Self-consistency improves chain of thought reasoning in language models , author =

  55. [60]

    Advances in Neural Information Processing Systems , publisher =

    Language Models are Few-Shot Learners , author =. Advances in Neural Information Processing Systems , publisher =

  56. [61]

    Proceedings of the 40th International Conference on Machine Learning , location =

    M\". Proceedings of the 40th International Conference on Machine Learning , location =

  57. [63]

    2602.06098 , archiveprefix =

    Coding Agents with Environment Interaction: A Theoretical Perspective , author =. 2602.06098 , archiveprefix =

    Pith/arXiv arXiv

  58. [64]

    2202.12837 , archiveprefix =

    Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? , author =. 2202.12837 , archiveprefix =

    Pith/arXiv arXiv

  59. [65]

    2311.00237 , archiveprefix =

    The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis , author =. 2311.00237 , archiveprefix =

    Pith/arXiv arXiv

  60. [66]

    An Explanation of In-context Learning as Implicit

    Sang Michael Xie and Aditi Raghunathan and Percy Liang and Tengyu Ma , year = 2022, booktitle =. An Explanation of In-context Learning as Implicit

    2022

  61. [67]

    2211.15661 , archiveprefix =

    What learning algorithm is in-context learning? Investigations with linear models , author =. 2211.15661 , archiveprefix =

    Pith/arXiv arXiv

  62. [68]

    2406.02543 , archiveprefix =

    To Believe or Not to Believe Your LLM , author =. 2406.02543 , archiveprefix =

    Pith/arXiv arXiv

  63. [70]

    2601.19897 , archiveprefix =

    Self-Distillation Enables Continual Learning , author =. 2601.19897 , archiveprefix =

    Pith/arXiv arXiv

  64. [71]

    2506.01876 , archiveprefix =

    In-Context Learning for Pure Exploration , author =. 2506.01876 , archiveprefix =

    arXiv

  65. [72]

    2509.24510 , archiveprefix =

    Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models , author =. 2509.24510 , archiveprefix =

    arXiv

  66. [73]

    2306.09927 , archiveprefix =

    Trained Transformers Learn Linear Models In-Context , author =. 2306.09927 , archiveprefix =

    Pith/arXiv arXiv

  67. [74]

    2212.07677 , archiveprefix =

    Transformers learn in-context by gradient descent , author =. 2212.07677 , archiveprefix =

    Pith/arXiv arXiv

  68. [75]

    Evaluating language models as risk scores , author =

  69. [76]

    Aligning Language Models from User Interactions , author =

  70. [77]

    2509.22638 , archiveprefix =

    Language Models Can Learn from Verbal Feedback Without Scalar Rewards , author =. 2509.22638 , archiveprefix =

    arXiv

  71. [78]

    Synthetic personas distort the structure of human belief systems , author =

  72. [79]

    2402.01908 , archiveprefix =

    Large language models that replace human participants can harmfully misportray and flatten identity groups , author =. 2402.01908 , archiveprefix =

    Pith/arXiv arXiv

  73. [80]

    Ang Li and Haozhe Chen and Hongseok Namkoong and Tianyi Peng , year = 2025, booktitle =

    2025

  74. [81]

    2502.16761 , archiveprefix =

    Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions , author =. 2502.16761 , archiveprefix =

    Pith/arXiv arXiv

  75. [82]

    2502.07068 , archiveprefix =

    Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations , author =. 2502.07068 , archiveprefix =

    Pith/arXiv arXiv

  76. [83]

    2603.06612 , archiveprefix =

    Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness , author =. 2603.06612 , archiveprefix =

    arXiv

  77. [84]

    2603.13305 , archiveprefix =

    Evidence-based Distributional Alignment for Large Language Models , author =. 2603.13305 , archiveprefix =

    arXiv

  78. [85]

    It's Time to Optimize for Self-Consistency , author =

  79. [86]

    2402.10946 , archiveprefix =

    CultureLLM: Incorporating Cultural Differences into Large Language Models , author =. 2402.10946 , archiveprefix =

    Pith/arXiv arXiv

  80. [87]

    Arik Reuter and Tim G. J. Rudner and Vincent Fortuin and David R. Can Transformers Learn Full

Showing first 80 references.