REVIEW 4 major objections 5 minor 10 references
Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The Guided Persona-based AI Survey reproduces the MiD 2017 mobility-preference distributions almost exactly, with MAE 0.03, RMSE 0.17, JS distance 0.0016, and Cramér's V 1.00.
desk verdict The guided persona method's near-perfect scores likely come from feeding the benchmark statistics into the prompt; the paper's comparison of the other five methods is the salvageable part. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Persona, a tuple $P(A,E,M,S,H)$ fixing an age group, education level, main activity, economic status, and household type. Each Persona's population share is built from a chain of conditional probabilities, $\pi_P = P_A(a)\cdot P_E(e\mid a)\cdot P_M(m\mid a,e)\cdot P_S(s\mid a,e,m)\cdot P_H(h\mid a,e,m,s)$, and the LLM's per-Persona responses are reweighted by those shares to form the synthetic survey. The Guided variant additionally inserts the MiD 2017 expected response statistics into the prompt, so the model is given the target answer averages while it simulates the 15,840 Personas; the evaluated output is the weighted aggregate, and it is that prompting step that separates the near-perfect scores from the weaker baselines.
What would settle it
Run the Guided Persona prompt with the MiD 2017 expected response statistics removed or corrupted, keeping all other settings fixed; if MAE, RMSE, and Cramér's V collapse toward the Structured Persona baseline, the near-perfect numbers are an artifact of reciting the prompt's provided answers, not evidence that the LLM learned demographic-response dependencies.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that an LLM-based generator reaches near-perfect statistical agreement with a real national travel survey when responses are produced per Persona and the prompt carries the survey's expected response distributions. The reported metrics — MAE 0.03, RMSE 0.17, JS distance 0.0016, Cramér's V 1.00 — are interpreted as the LLM capturing complex dependencies between demographic attributes and mobility preferences, not as a reproduction of the supplied statistics. The comparison against five lower-performing baselines is used to argue that both the persona structure and the guided constraints are necessary for the fidelity.
Load-bearing premise
The method's success depends on treating the insertion of the survey's expected response statistics into the LLM prompt as part of the model's learning, rather than as the answer key being handed to the model.
Editorial extensions
If this is right
- Synthetic mobility surveys can be produced at scale, as the paper generates 10,000-individual populations and 15,840 Personas from prompt-level specifications.
- Because only aggregate benchmark statistics are needed, the method offers a privacy-preserving way to study population-level mobility preferences without individual records.
- The same recipe — persona attributes, conditional shares, and guided prompting with expected statistics — should transfer to other transport modes, other countries, and other survey years.
- The large error drop between guided and non-guided variants implies that response-level constraints, not demographic alignment alone, drive distributional fidelity.
- The evaluation kit (MAE, RMSE, JS distance, entropy, Cramér's V) provides a reusable standard for judging whether any synthetic survey is statistically interchangeable with a real one.
Reading between the lines
- The near-perfect metrics are likely measuring prompt fidelity, not learned demographic reasoning, since the benchmark's expected response statistics are already inside the winning prompt; withholding them would reveal what the LLM can actually infer.
- A direct extension would be a held-out test: guide the model with MiD 2017 statistics for some questions, ask for other questions never summarized, and see whether distributional accuracy survives.
- The persona conditional-probability chain could generate a synthetic survey without any LLM, so the paper does not isolate the LLM's added value; a sampling-only baseline over the same shares would clarify that.
- Transport planners could use the guided method for counterfactual scenarios, such as changing household mixes or economic-status distributions, but those counterfactuals inherit the assumption that the LLM's response profiles, not just the provided shares, are trustworthy.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a 'Guided Persona-based AI Survey' method in which GPT-4o generates synthetic mobility-survey responses for a German population, and compares this method against five alternatives using the MiD 2017 mobility survey as a benchmark. The paper reports that the Guided Persona-based method achieves near-perfect alignment with MiD 2017 (MAE=0.03, RMSE=0.17, JS Distance=0.0016, Cramér's V=1.00) and interprets this as evidence that LLMs can effectively capture complex dependencies between demographic attributes and preferences, with implications for scalable, privacy-preserving synthetic survey generation.
Significance. If the reported result were valid, the paper would make a useful contribution to synthetic survey generation for transportation and social science research: it clearly defines six generation methods, uses standard distribution-comparison metrics, and makes an effort toward reproducibility by promising a GitHub repository with prompts and code. However, the central evidence is undermined by a circular evaluation design visible in the manuscript itself: the Guided Persona-based method is given MiD 2017 expected response statistics as input and is then scored against the same MiD 2017 dataset, so the near-perfect metrics do not demonstrate any independent learning or generation capability. The paper provides no held-out validation, no repeated trials, no uncertainty quantification, and no explicit prompt text that would allow the reader to assess what information was leaked into the model context.
major comments (4)
- [Section 2, 'Guided Persona-based AI Survey', and Table 1] The headline result is circular. The method is described as 'incorporated expected response statistics from the MiD 2017 dataset', and the evaluation in Table 1 compares the output of that method against the same MiD 2017 dataset. The near-perfect metrics (MAE=0.03, RMSE=0.17, JS Distance=0.0016, Cramér's V=1.00) are precisely what would be expected if the model re-emits the supplied tabulated statistics. The passage in Section 2 instructing the model to 'maintain realistic correlations ... based on MiD 2017 data' confirms that the benchmark answers were not withheld. The authors should report an evaluation on a held-out split of MiD 2017, a no-leakage ablation in which the target statistics are removed from the prompt, and a baseline that simply re-weights the prompted statistics without an LLM; without at least one of these, the central claim that LLMs 'effectively capture complex dependencies' is unsupported.
- [Section 2, Eqs. (1)-(4)] The Persona construction is a deterministic product of conditional probabilities and a weighted sum. If the conditional probabilities PA, PE, PM, PS, PH in Eq. (2) are estimated from MiD 2017, then the pipeline can reproduce the MiD 2017 marginals by construction, and the LLM contributes nothing except surface form. The paper does not state where these conditional probabilities come from or how they differ across the Persona-based methods. The authors should clarify the provenance of every input distribution and demonstrate that the observed accuracy is not a foregone mathematical consequence of Eq. (2).
- [Section 3, 'Evaluation Metrics' and Table 1] Only a single generation is reported for each method. GPT-4o is stochastic, and no temperature setting, random seed, number of repeated runs, or confidence interval is given. The reported Cramér's V = 1.00 and MAE = 0.03 would be more credible with variance estimates across repeated generations; without them, the reader cannot distinguish a stable property of the method from a single favorable draw.
- [Section 2, 'Survey Generation Methods'] The full prompts are only available at an external GitHub repository, not in the manuscript. Since the central question is what information was placed in the context, the paper should quote the exact Guided Persona prompt in the text or appendix; the current brief quotation is insufficient to rule out that the model was asked to output the provided MiD 2017 statistics.
minor comments (5)
- [Section 1] The claim that the guided Persona-based approach is 'novel' and 'to the best of our knowledge, has not been previously applied' would benefit from a more systematic comparison to the cited synthetic-data literature; at present the related work is described in a single paragraph.
- [Section 2, 'Dataset Overview'] 'The percentage of "not specified" option is merged to the most popular responses' is grammatically unclear and should specify whether the merge was done before or after normalization.
- [Section 3, 'Key Findings'] The phrase 'lower JS Distance scores' in the fourth bullet is confusing because lower JS Distance is better; the sentence appears to describe 'limited alignment' while citing a lower error value.
- [Figure 3] The subplots are referenced in the text as (a)-(g), but the caption does not define the color scheme; a legend or an explicit statement that grey lines are real survey data is needed.
- [Table 1] The metric '|Conditional Entropy|' is presented as an absolute value without explanation of why the sign is relevant or why the absolute value is taken.
Circularity Check
The Guided Persona-based AI Survey's near-perfect scores are circular: MiD 2017 expected response statistics are inserted into the prompt and then the same MiD 2017 data is used as the benchmark, so the reported 'prediction' is largely a re-read of the input.
-
fitted input called prediction
[Section 2, Survey Generation Methods, Guided Persona-based AI Survey bullet]
"Guided Persona-based AI Survey: Building upon the Structured Persona-based AI Survey, this method incorporated expected response statistics from the MiD 2017 dataset."
The method's defining input is the MiD 2017 expected response statistics, and Table 1 then evaluates the same method against MiD 2017, reporting MAE=0.03 and Cramer's V=1.00. The 'prediction' is not independent of the benchmark; it is the benchmark statistics being re-emitted through the LLM prompt.
-
fitted input called prediction
[Section 2, Survey Generation Methods, prompt description paragraph]
"advanced directives that explicitly instructed the LLM to consider correlations between attributes (e.g. "Simulate mobility preferences while maintaining realistic correlations between age, household type and economic status based on MiD 2017 data")."
The prompt explicitly instructs the model to reproduce MiD 2017 correlations, and the evaluation then measures how well the synthetic data reproduces those same correlations. Scoring the model against the exact correlations it was told to maintain is circular: the target is part of the input.
1 more flagged steps
-
self definitional
[Section 2, Persona Methods, Eqs. (2) and (4)]
"πP = PA(a) · PE(e | a) · PM (m | a, e) · PS(s | a, e, m) · PH (h | a, e, m, s) ... Ri(a) = X P ∈A=a πP · Ri(P)"
If the conditional probabilities and Persona response probabilities Ri(P) are taken from MiD 2017 expected statistics, as the method description states, then the weighted response Ri(a) is a deterministic recomputation of MiD response rates. The formal equations define the output directly from the benchmark input, so the near-perfect alignment is by construction rather than evidence of LLM-learned dependencies.
full rationale
The paper's central claim is that the Guided Persona-based AI Survey shows LLMs can 'effectively capture complex dependencies between demographic attributes and preferences.' The reported evidence does not support that claim because the method is circular by construction. Section 2 states that the Guided Persona-based AI Survey 'incorporated expected response statistics from the MiD 2017 dataset,' and Table 1 scores that method against the same MiD 2017 dataset. The prompt description confirms that the model was explicitly told to maintain correlations 'based on MiD 2017 data.' The formal Persona construction in Eqs. (1)-(4) is a weighted sum of conditional probabilities and response probabilities; when those probabilities are the MiD 2017 statistics, the output distribution is a deterministic recombination of the benchmark itself. The near-perfect metrics (MAE=0.03, RMSE=0.17, JS Distance=0.0016, Cramer's V=1.00) are exactly what one would expect from an LLM reading tabulated statistics back from the context, not from emergent inference. The paper does not report a no-leakage ablation, a held-out subset, repeated runs with different seeds, or a baseline that simply tabulates the prompted statistics, so the leakage hypothesis cannot be dismissed. This is not a self-citation issue; it is an input-to-prediction reduction. The score is 8 rather than 10 because the paper is transparent about the 'guided' nature of its method, but the central evaluative claim is still forced by the benchmark statistics being placed in the prompt.
Assumptions & free parameters
assumptions (3)
- domain assumption MiD 2017 is an accurate ground truth for mobility preferences and can be treated as the true population distribution.
- domain assumption The chain rule decomposition in Eq. (2) correctly captures all significant correlations among the five demographic attributes.
- domain assumption LLM responses conditioned on persona attributes and target statistics approximate human survey responses.
Cite this review
Pith. "Pith review of Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?." pith.science (2026). https://pith.science/paper/E7P5SMS7
@misc{pith2026250113955,
author = {Pith},
title = {Pith review of: Guided Persona-based AI Surveys: Can we replicate personal mobility preferences at scale using LLMs?},
year = {2026},
howpublished = {\url{https://pith.science/paper/E7P5SMS7}},
note = {Machine review of arXiv:2501.13955}
}
read the original abstract
This study explores the potential of Large Language Models (LLMs) to generate artificial surveys, with a focus on personal mobility preferences in Germany. By leveraging LLMs for synthetic data creation, we aim to address the limitations of traditional survey methods, such as high costs, inefficiency and scalability challenges. A novel approach incorporating "Personas" - combinations of demographic and behavioural attributes - is introduced and compared to five other synthetic survey methods, which vary in their use of real-world data and methodological complexity. The MiD 2017 dataset, a comprehensive mobility survey in Germany, serves as a benchmark to assess the alignment of synthetic data with real-world patterns. The results demonstrate that LLMs can effectively capture complex dependencies between demographic attributes and preferences while offering flexibility to explore hypothetical scenarios. This approach presents valuable opportunities for transportation planning and social science research, enabling scalable, cost-efficient and privacy-preserving data generation.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Bisbee_Clinton_Dorff_Kenkel_Larson_2024 APACrefauthors Bisbee, J. , Clinton, J D. , Dorff, C. , Kenkel, B. \ Larson, J M. APACrefauthors \ 2024 . Synthetic Replacements for Human Survey Data? The Perils of Large Language Models Synthetic replacements for human survey data? the perils of large language models . Political Analysis 32 4 401–416 . APACrefDOI ...
-
[2]
10758652 APACrefauthors Cho, S. , Kim, J. \ Kim, J H. APACrefauthors \ 2024 . LLM-Based Doppelgänger Models: Leveraging Synthetic Data for Human-Like Responses in Survey Simulations Llm-based doppelgänger models: Leveraging synthetic data for human-like responses in survey simulations . IEEE Access 12 178917-178927 . APACrefDOI doi:10.1109/ACCESS.2024.350...
arXiv 2024
-
[3]
10366424 APACrefauthors Choenni, S. , Busker, T. \ Bargh, M S. APACrefauthors \ 2023 . Generating Synthetic Data from Large Language Models Generating synthetic data from large language models . 2023 15th International Conference on Innovations in Information Technology (IIT) 2023 15th international conference on innovations in information technology (iit...
-
[4]
liu2024bestpracticeslessonslearned APACrefauthors Liu, R. , Wei, J. , Liu, F. , Si, C. , Zhang, Y. , Rao, J. Dai, A M. APACrefauthors \ 2024 . Best Practices and Lessons Learned on Synthetic Data. Best practices and lessons learned on synthetic data. APACrefURL https://arxiv.org/abs/2404.07503 APACrefURL
arXiv 2024
-
[5]
livieris2024evaluationframeworksyntheticdata APACrefauthors Livieris, I E. , Alimpertis, N. , Domalis, G. \ Tsakalidis, D. APACrefauthors \ 2024 . An evaluation framework for synthetic data generation models. An evaluation framework for synthetic data generation models. APACrefURL https://arxiv.org/abs/2404.08866 APACrefURL
-
[6]
ma2024algorithmicfidelitylargelanguage APACrefauthors Ma, B. , Yoztyurk, B. , Haensch, A C. , Wang, X. , Herklotz, M. , Kreuter, F. Assenmacher, M. APACrefauthors \ 2024 . Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study. Algorithmic fidelity of large language models in generating synthetic german ...
work page Pith review arXiv 2024
-
[7]
mid2017 APACrefauthors MiD. APACrefauthors \ 2017 . MiD 2017 Tabellenband Deutschland. Mid 2017 tabellenband deutschland. APACrefURL https://www.mobilitaet-in-deutschland.de/archive/pdf/MiD2017_Tabellenband_Deutschland.pdf APACrefURL
work page 2017
-
[8]
tan2024largelanguagemodelsdata APACrefauthors Tan, Z. , Li, D. , Wang, S. , Beigi, A. , Jiang, B. , Bhattacharjee, A. Liu, H. APACrefauthors \ 2024 . Large Language Models for Data Annotation and Synthesis: A Survey. Large language models for data annotation and synthesis: A survey. APACrefURL https://arxiv.org/abs/2402.13446 APACrefURL
arXiv 2024
Show all 10 references
-
[9]
APACrefauthors \ 2024
tzachristas2024creatingllmbasedaiagenthighlevel APACrefauthors Tzachristas, I. APACrefauthors \ 2024 . Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs. Creating an llm-based ai-agent: A high-level methodology towards enhancing llms wit...
2024 arXiv
-
[10]
, Zhu, J
wang2024surveydatasynthesisaugmentation APACrefauthors Wang, K. , Zhu, J. , Ren, M. , Liu, Z. , Li, S. , Zhang, Z. Wang, Y. APACrefauthors \ 2024 . A Survey on Data Synthesis and Augmentation for Large Language Models. A survey on data synthesis and augmentation for large lang...
2024 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.