Pith. sign in

REVIEW 3 major objections 5 references

LLMs can generate synthetic projective consumer responses that cover similar broad topics and diversity levels as humans, while still differing in style and how that diversity is produced.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 02:21 UTC pith:LTR3EXE3

load-bearing objection Solid multi-factor benchmarking of LLM projective data against a real tourism study; useful practice guidance, but human–LLM similarity stays mostly descriptive. the 3 major comments →

arxiv 2607.05761 v1 pith:LTR3EXE3 submitted 2026-07-07 cs.AI

Synthetic Consumer Insight Generation with Large Language Models

Stephen L. France , Pia. A. Albinsson This is my paper
classification cs.AI
keywords Synthetic Data GenerationLarge Language ModelsProjective TechniquesTopic ModelingMarketing ResearchConsumer InsightsPrompt EngineeringAgentic Information Systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether large language models can generate usable synthetic answers to projective marketing tasks—word association and positive/negative sentence completion—that marketers use to surface associations, emotions, and latent needs. Using a primary study of college-student perceptions of five U.S. tourism cities as the human benchmark, the authors systematically vary models, temperature, and prompting strategies (basic, extended, verbalized sampling, and few-shot seeding with real answers) and compare outputs with linguistic, diversity, concentration, topic-model, and top-term analyses. They find substantial overlap in broad topics and associations, so LLMs can recover many of the same city meanings humans produce. At the same time, LLM answers are often more verbose and polished, and they often create lexical diversity through compound adjectives and stylized phrases rather than the same mix of brief, lived associations. The paper therefore treats synthetic projective data as a practical, low-cost tool for idea generation, piloting, and expanding insight—not as a substitute for estimating how common a belief is in a real population—and gives concrete guidance on how model and prompt choices shape quality.

Core claim

By controlling prompting strategy and temperature, LLMs can generate projective consumer responses with similar diversity characteristics and covering similar broad topics and associations to human responses, while still differing in style, linguistic structure, and the mechanism by which diversity is generated.

What carries the argument

A multi-factor generation-and-evaluation design that crosses LLM, temperature, and prompting strategy (basic, extended, verbalized sampling, 10-/50-shot human seeds) and scores the outputs with length/style metrics, diversity and concentration indices, structural topic models, and city-specific term lift against a fixed human projective corpus.

Load-bearing premise

That a single Southeast U.S. college-student sample plus a matching “Imagine you are a {gender} college student from the South East” persona is an adequate human benchmark for deciding whether synthetic projective data give equivalent consumer insight.

What would settle it

Collect the same projective tasks from a demographically different human sample (for example older leisure travelers) and test whether the LLM settings that best matched the student benchmark still match that sample’s topics, diversity metrics, and top terms—or systematically diverge.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • LLMs can serve as a low-cost supplement for idea generation, piloting, and expanding projective insight when budgets or timelines are tight.
  • Extended prompts and higher temperature raise diversity; few-shot human examples can pull content and style closer to a target human set.
  • Model choice strongly shapes verbosity, vocabulary breadth, and concentration, so practitioners must tune models to the intended use.
  • Synthetic answers should not be used to claim population frequencies of beliefs or perceptions.
  • Even when broad topics match, researchers should expect more polished, compound, and stylized phrasing than typical human projective answers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same protocol could be stress-tested on longer construction tasks (brand stories, scenario essays), where stylistic polish may either help creativity or further distance the text from lived experience.
  • Because LLM diversity often arrives via compound neologisms, validation pipelines may need a separate human-likeness-of-phrasing check beyond topic coverage and entropy.
  • Segment-specific personas beyond a single student profile would be a natural next control if the goal is segment insight rather than a generic student mirror.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper asks whether LLMs can generate synthetic consumer responses for projective techniques (word association and positive/negative sentence completion) that resemble human responses and yield comparable insights. Using a human benchmark of n≈173 Southeast U.S. college students on five U.S. tourism cities, the authors generate matched synthetic corpora across six LLMs, five temperatures, and five prompting strategies (basic, extended, verbalized sampling, and 10-/50-shot). They compare outputs via linguistic and diversity/concentration metrics (Tables 5–6), scatterplots of human vs. LLM aggregates (Figs. 2–7), structural topic models (Tables 7–8), and lift-based top terms (Figs. 10–12). The central claim is that, with prompt and temperature control, LLMs can match broad diversity characteristics and topical associations while differing in style, linguistic structure, and the mechanism of diversity generation; managerial recommendations follow from that claim.

Significance. If the claim holds under a clearer equivalence standard, the paper would be a useful contribution at the marketing/IS interface: it moves synthetic-consumer work beyond structured choice and survey items into open-ended projective tasks, provides a multi-model factorial design with explicit prompts, and pairs distributional metrics with topic and lift analyses that reveal how LLM diversity is often produced via compound adjectives rather than new themes. The practical framing—LLMs as low-cost idea generation and piloting tools, not substitutes for prevalence estimation—is appropriately cautious and actionable. Strengths include transparent experimental factors, substantial R² on condition-level regressions, and content-level diagnostics that go beyond surface fluency.

major comments (3)
  1. Discussion RQ1 / Abstract: the load-bearing claim that LLMs produce responses with 'similar diversity characteristics' and 'substantial overlap… in broad topics and associations' (and 'equivalent insights') is supported almost entirely by descriptive corpus aggregates. The paper itself states that corpus-level metrics yield one value per condition and are 'primarily descriptive rather than inferential' (Basic Linguistic Characteristics). Figs. 2–7 plot single human points against LLM aggregates; Tables 7–8 and Figs. 10–12 are qualitative. There is no formal human–LLM difference test, no document-level topic-prevalence comparison, and no held-out check that human topic structure is recovered by LLM text. Without at least one inferential or predictive bridge, 'resemble' and 'equivalent insights' remain under-supported relative to the abstract and recommendations.
  2. Methodology (human sample) and Basic prompt: the human benchmark is a single-university Southeast U.S. student sample (n≈173), and the persona is fixed as '{gender} college student from the South East of the U.S.' This is a legitimate design choice for a tourism-student context, but the paper generalizes to 'human participants' and 'synthetic consumer data' without bounding external validity. The claim that synthetic data can give equivalent insights needs either multi-sample validation or explicit scope limits in the abstract, discussion, and managerial implications.
  3. Case Study: Topic Modeling: STM is estimated separately for human and one LLM condition (Gemini 2.5, ExtendedProb, T=1.8), each with K=9 chosen partly for matching convenience. Concordance is asserted by side-by-side topic labels (Tables 7–8) and lift lists (Figs. 10–12), but there is no joint model, no alignment metric (e.g., topic-word cosine / Hungarian matching), and no prevalence comparison by city. The important observation that LLM diversity often comes from compound adjectives is insightful but currently qualitative; a quantitative comparison of multiword-token rates or topic exclusivity would make the 'similar topics, different diversity mechanism' claim load-bearing rather than impressionistic.

Circularity Check

0 steps flagged

Empirical human–LLM benchmarking with external human responses; no derivation that reduces to its own inputs by construction.

full rationale

This paper is an empirical comparison study, not a first-principles derivation. Human projective responses (n≈173 college students) serve as an external benchmark; LLM outputs are generated under stated models, temperatures, and prompts, then compared via standard linguistic, diversity/concentration, topic-model, and lift analyses. The central claim—that with prompt and temperature control LLMs can produce responses with similar diversity characteristics and broad topic overlap, while differing in style and how diversity is produced—is supported by descriptive corpus metrics and qualitative topic/term tables, not by fitting a parameter and re-labeling it as a prediction. Few-shot conditions (ExtendedProb10/50Shot) intentionally inject human examples, but the paper treats them as experimental design choices and also reports non-seeded Basic and ExtendedProb conditions; similarity claims are not forced solely by those seeds. The Blinded Authors (2025) citation supplies the human dataset and is not a load-bearing uniqueness theorem or ansatz. No equation equates a fitted quantity to a claimed prediction by construction. Score 0 is therefore appropriate: the work is self-contained against an external human benchmark, with no circular reduction of the kind the analyzer targets.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on experimental design choices and domain assumptions rather than invented physical entities. Load-bearing free parameters are sampling and analysis knobs (temperature grid, Mistral rescaling, topic count, lift smoothing). Domain axioms include the adequacy of projective tasks and the student tourism sample as a human ground truth. No new theoretical objects are postulated.

free parameters (5)
  • LLM temperature grid = 1.0, 1.2, 1.4, 1.6, 1.8
    Temperatures 1.0–1.8 were chosen after pretests; values below 1.0 were nearly deterministic and above ~1.8 produced unacceptable degeneracy. These hand-chosen bounds shape measured diversity.
  • Mistral temperature multiplier = 0.75
    Mistral temperature was multiplied by 0.75 due to a lower effective upper bound; this ad hoc rescaling affects cross-model diversity comparisons.
  • Topic count K for STM = 9
    Nine topics were selected via diagnostics and face validity for both human and selected LLM corpora; topic concordance claims depend on this choice.
  • Lift smoothing alpha = 0.5
    Additive smoothing α=0.5 in the city-term lift formula prevents zeros and affects top-term rankings used for content comparison.
  • Responses per condition r = 173
    r=173 was matched to the human sample size; corpus-level diversity metrics are sample-size sensitive.
axioms (5)
  • domain assumption Projective word-association and sentence-completion tasks are valid instruments for eliciting consumer associations and latent perceptions comparable across human and LLM respondents.
    Invoked throughout Introduction and Methodology as the methodological testbed; validity of the human–LLM comparison depends on this.
  • domain assumption A Southeast U.S. college-student sample’s city perceptions are an appropriate human benchmark for the synthetic-data evaluation.
    Methodology justifies the sample as salient for leisure travel/post-graduation relocation; general claims about “human participants” rest on this.
  • domain assumption Standard lexical diversity/concentration metrics (entropy, Simpson, Yule’s K, hapax, TTR) and STM topic models adequately capture interpretive richness of projective responses.
    Main Results define and regress these metrics as primary evaluation axes for similarity and diversity.
  • ad hoc to paper Holding top-p at defaults while varying temperature is sufficient to study response diversity without confounding.
    Methodology cites OpenAI guidance to vary one of temperature or top-p; alternative decoding regimes are not tested.
  • standard math Tokenization, stopword lists, and stemming preserve the substantive content needed for cross-condition comparison.
    Standard NLP preprocessing assumptions used before metric and topic analyses (Main Results).

pith-pipeline@v1.1.0-grok45 · 29341 in / 3402 out tokens · 41257 ms · 2026-07-11T02:21:21.519263+00:00 · methodology

0 comments
read the original abstract

Modern data-driven marketing relies on large amounts of consumer data, yet collecting such data can be costly, time-consuming, and difficult to scale. This research examines whether large language models (LLMs) can be used to generate synthetic consumer data for projective techniques, a set of methods designed to elicit consumer associations, emotions, wants, and needs. We test LLM-generated responses across multiple projective tasks, LLMs, prompting strategies, and temperature settings, and compare them with human responses from a primary research study on perceptions of city tourism destinations. Human and LLM responses were analyzed using linguistic measures, diversity and concentration metrics, topic models, and top-term analyses. The results show substantial overlap between human and LLM responses in broad topics and associations, but also important differences in style, linguistic structure, and the way diversity is generated. Recommendations are given on how to best utilize LLMs for generating synthetic consumer data, how model and prompt choices shape response quality, and on recognizing the limitations of LLM synthetic consumer data generation.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 5 canonical work pages · 4 internal anchors

  1. [1]

    Abramova, O., Batzel, K., & Modesti, D. (2022). Collective response to the health crisis among German Twitter users: a structural topic modeling approach. International Journal of Information Management Data Insights, 2(2), 100126. Alencar, N. M. M., de Araújo, V. A., Faggian, L., da Silveira Araújo, M. B., & Capriles, V. D. (2021). What about gluten‐free...

    work page 2022

  2. [2]

    Boddy, C. R. (2005). Projective techniques in market research: valueless subjectivity or insightful reality? A look at the evidence for the usefulness, reliability and validity of projective techniques in market research. International Journal of Market Research, 47(3), 239-254. Bond, D., & Ramsey, E. (2010). The role of information and communication tech...

    work page internal anchor Pith review Pith/arXiv arXiv 2005

  3. [3]

    Hindley, A., & Font, X. (2018). The use of projective techniques to circumvent socially desirable responses or reveal the subconscious. In R. Nunkoo (Ed.), Handbook of research methods for tourism and hospitality management (pp. 202-210). Elgar. 46 Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The curious case of neural text degeneration....

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  4. [4]

    Large Language Model Routing with Benchmark Datasets

    Sarstedt, M., Adler, S. J., Rau, L., & Schmitt, B. (2024). Using large language models to generate silicon samples in consumer and marketing research: Challenges, opportunities, and guidelines. Psychology & Marketing, 41(6), 1254-1270. Sarstedt, M., Adler, S. J., Rau, L., & Schmitt, B. (2026). Your Next Respondent Might Be an LLM: Guidelines for Using Sil...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Simpson, E. H. (1949). Measurement of diversity. Nature, 163(4148), 688-688. Singh, J., & Gupta, V. (2016). Text stemming: Approaches, applications, and challenges. ACM Computing Surveys (CSUR), 49(3), 1-46. Smolyak, D., Bjarnadóttir, M. V., Crowley, K., & Agarwal, R. (2024). Large language models and synthetic health data: progress and prospects. JAMIA O...

    work page internal anchor Pith review Pith/arXiv arXiv 1949