{"id":"890f1c2a-103c-4d99-a8d3-16d8e3c4bcd0","arxiv_id":"2501.13955","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A persona-based LLM survey generator given MiD 2017 response statistics reproduces those statistics almost exactly, but the evaluation is circular and the main claim is unsupported.","lead":"This paper tests whether LLMs can generate realistic synthetic mobility surveys by comparing six prompting strategies against Germany's MiD 2017 dataset. Its headline result, near-perfect replication by a 'guided persona' method, is likely an artifact of feeding the benchmark statistics into the model.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-perfect metrics for the Guided Persona-based AI Survey are likely an artifact of placing MiD 2017 response statistics inside the prompt and then scoring against the same MiD 2017 data; the paper's central claim is not supported by the reported evidence.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the Guided Persona-based method is evaluated against the same dataset that supplied the prompt statistics, so the near-perfect metrics are likely circular. My review of the full text confirms this. Section 2 explicitly states that the Guided Persona-based AI Survey 'incorporated expected response statistics from the MiD 2017 dataset' and that prompts instructed the model to consider correlations 'based on MiD 2017 data.' Section 3 then scores the method against MiD 2017. The formal definitions in Eqs. (1)-(4) show that a Persona-based weighted sum can be constructed from conditional probabilities, so if those probabilities are the target dataset's statistics, the method can reach perfect alignment without the LLM displaying any real understanding of mobility behavior. The paper also does not include the actual prompt text, does not report hold-out evaluation, and does not include a baseline that directly uses the prompted statistics without an LLM, all of which would be necessary to distinguish genuine dependency learning from answer recall. I agree with the REJECT verdict, because the central scientific claim is not supported by the reported evidence. The concern is not about author intent; it is about experimental design and the need for a leakage-free evaluation. A concrete ablation of the prompt contents, or an independent reimplementation of the weighted-sum formula, would settle the issue.","tokens_in":5188,"tokens_out":2721,"duration_ms":33801,"concrete_test":"Retrieve the exact Guided Persona-based AI Survey prompt from the linked GitHub repository and inspect whether the prompt contains numeric expected response statistics or conditional distributions derived from MiD 2017. Then run a redaction ablation: remove all MiD-derived response statistics from the prompt while keeping the demographic Persona structure and the instruction to model realistic correlations. If the redacted-prompt MAE remains near 0.03, the original result supports genuine dependency learning; if it jumps to roughly the Structured Persona-based level (MAE around 8.5), the reported near-perfect alignment is explained by prompt leakage. As a second confirmatory check, implement Eqs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the Guided Persona-based AI Survey shows LLMs can 'effectively capture complex dependencies between demographic attributes and preferences.' The decisive problem is that the method is described in Section 2 as incorporating 'expected response statistics from the MiD 2017 dataset' into the prompt, and then Table 1 evaluates that same method against MiD 2017. This is a train-on-test setup: the benchmark answer key is part of the model's input context. The reported MAE=0.03, RMSE=0.17, JS Distance=0.0016, and Cramér's V=1.00 are exactly what one would expect if the LLM re-emitted tabulated statistics from the prompt rather than inferring demographic-response dependencies. The paper even quotes an example prompt instruction to 'maintain realistic correlations ... based on MiD 2017 data,' which reinforces that the target statistics were not withheld. Moreover, the formal Persona construction in Eqs. (1)-(4) is a deterministic weighted sum over conditional probabilities; if those conditional probabilities are taken from MiD 2017, the pipeline could reproduce the benchmark without any LLM-generated behavioral content. The authors do not report a no-leakage ablation, a held-out subset, an independent baseline using just the prompted statistics, or repeated runs with different random seeds. The GitHub repository is promised as the source of the exact prompts, but the manuscript itself does not show the prompt text, so the leakage hypothesis cannot currently be checked from the paper. Thus the near-perfect alignment does not provide evidence for the claim that LLMs capture complex dependencies; it may merely demonstrate that the model echoes supplied numbers. The comparison among the other five methods may be informative, but it is not the paper's headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'Guided Persona-based AI Survey' method in which GPT-4o generates synthetic mobility-survey responses for a German population, and compares this method against five alternatives using the MiD 2017 mobility survey as a benchmark. The paper reports that the Guided Persona-based method achieves near-perfect alignment with MiD 2017 (MAE=0.03, RMSE=0.17, JS Distance=0.0016, Cramér's V=1.00) and interprets this as evidence that LLMs can effectively capture complex dependencies between demographic attributes and preferences, with implications for scalable, privacy-preserving synthetic survey generation.","tokens_in":5427,"tokens_out":4081,"duration_ms":40568,"significance":"If the reported result were valid, the paper would make a useful contribution to synthetic survey generation for transportation and social science research: it clearly defines six generation methods, uses standard distribution-comparison metrics, and makes an effort toward reproducibility by promising a GitHub repository with prompts and code. However, the central evidence is undermined by a circular evaluation design visible in the manuscript itself: the Guided Persona-based method is given MiD 2017 expected response statistics as input and is then scored against the same MiD 2017 dataset, so the near-perfect metrics do not demonstrate any independent learning or generation capability. The paper provides no held-out validation, no repeated trials, no uncertainty quantification, and no explicit prompt text that would allow the reader to assess what information was leaked into the model context.","major_comments":[{"comment":"The headline result is circular. The method is described as 'incorporated expected response statistics from the MiD 2017 dataset', and the evaluation in Table 1 compares the output of that method against the same MiD 2017 dataset. The near-perfect metrics (MAE=0.03, RMSE=0.17, JS Distance=0.0016, Cramér's V=1.00) are precisely what would be expected if the model re-emits the supplied tabulated statistics. The passage in Section 2 instructing the model to 'maintain realistic correlations ... based on MiD 2017 data' confirms that the benchmark answers were not withheld. The authors should report an evaluation on a held-out split of MiD 2017, a no-leakage ablation in which the target statistics are removed from the prompt, and a baseline that simply re-weights the prompted statistics without an LLM; without at least one of these, the central claim that LLMs 'effectively capture complex dependencies' is unsupported.","section":"Section 2, 'Guided Persona-based AI Survey', and Table 1"},{"comment":"The Persona construction is a deterministic product of conditional probabilities and a weighted sum. If the conditional probabilities PA, PE, PM, PS, PH in Eq. (2) are estimated from MiD 2017, then the pipeline can reproduce the MiD 2017 marginals by construction, and the LLM contributes nothing except surface form. The paper does not state where these conditional probabilities come from or how they differ across the Persona-based methods. The authors should clarify the provenance of every input distribution and demonstrate that the observed accuracy is not a foregone mathematical consequence of Eq. (2).","section":"Section 2, Eqs. (1)-(4)"},{"comment":"Only a single generation is reported for each method. GPT-4o is stochastic, and no temperature setting, random seed, number of repeated runs, or confidence interval is given. The reported Cramér's V = 1.00 and MAE = 0.03 would be more credible with variance estimates across repeated generations; without them, the reader cannot distinguish a stable property of the method from a single favorable draw.","section":"Section 3, 'Evaluation Metrics' and Table 1"},{"comment":"The full prompts are only available at an external GitHub repository, not in the manuscript. Since the central question is what information was placed in the context, the paper should quote the exact Guided Persona prompt in the text or appendix; the current brief quotation is insufficient to rule out that the model was asked to output the provided MiD 2017 statistics.","section":"Section 2, 'Survey Generation Methods'"}],"minor_comments":[{"comment":"The claim that the guided Persona-based approach is 'novel' and 'to the best of our knowledge, has not been previously applied' would benefit from a more systematic comparison to the cited synthetic-data literature; at present the related work is described in a single paragraph.","section":"Section 1"},{"comment":"'The percentage of \"not specified\" option is merged to the most popular responses' is grammatically unclear and should specify whether the merge was done before or after normalization.","section":"Section 2, 'Dataset Overview'"},{"comment":"The phrase 'lower JS Distance scores' in the fourth bullet is confusing because lower JS Distance is better; the sentence appears to describe 'limited alignment' while citing a lower error value.","section":"Section 3, 'Key Findings'"},{"comment":"The subplots are referenced in the text as (a)-(g), but the caption does not define the color scheme; a legend or an explicit statement that grey lines are real survey data is needed.","section":"Figure 3"},{"comment":"The metric '|Conditional Entropy|' is presented as an absolute value without explanation of why the sign is relevant or why the absolute value is taken.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"For the editor: the circularity is apparent from the manuscript itself in Section 2, so the central claim cannot be supported by the reported evidence. A revision that removes the leakage would amount to a new experimental design rather than a local repair, hence the reject recommendation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim doesn't survive contact with the method section. The Guided Persona-based AI Survey is described as incorporating expected response statistics from the MiD 2017 dataset into the prompt, and then it's scored against the same MiD 2017 data. MAE 0.03, JS 0.0016, Cramér's V 1.00 are exactly what you'd expect if the model re-emitted the supplied numbers. There's no held-out split, no repeated trials, no seed variation, and no comparison against a trivial baseline that just draws from the tabulated statistics. The example prompt quoted in Section 2 ('maintain realistic correlations ... based on MiD 2017 data') reinforces the leak. So the headline result is not evidence that LLMs capture demographic-response dependencies; it's evidence that GPT-4o can follow an instruction to copy a table.\n\nWhat's genuinely useful is the six-method comparison. The non-persona Guided AI Survey also gets the benchmark statistics as a constraint but ends up with MAE 3.37 and Cramér's V 0.68, so the persona construction and the way the constraints are phrased do matter. The formal persona definition in Equations (1)-(4) is a clean way to build weighted demographic combinations, and the paper is honest enough to include a naive baseline that doesn't use any real data. If the authors re-did the evaluation with a held-out portion of MiD 2017, or at least against a different wave or year, the comparison would be informative.\n\nThe soft spots are mostly about what's missing. No prompt text in the paper, no commit hash, no model version beyond 'GPT-4o API'. The GitHub link exists but the manuscript doesn't show enough to reproduce the decisive condition. The citation pattern is fine; Bisbee et al., Ma et al., and the other synthetic-data papers are the right ones to cite, and the self-citation is a methodology paper, not a problem.\n\nBottom line: the paper's own evidence contradicts its interpretation. The comparison among the other five methods is worth keeping, but as written the central claim is unsupported and the evaluation design needs a rework. If the authors can do a clean held-out version, this becomes a decent methods paper. For now, I'd tell a student to read it for the negative result, not for the headline.","headline":"The guided persona method's near-perfect scores likely come from feeding the benchmark statistics into the prompt; the paper's comparison of the other five methods is the salvageable part.","tokens_in":6047,"tokens_out":1744,"would_cite":false,"duration_ms":18299,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Guided Persona-based AI Survey reproduces the MiD 2017 mobility-preference distributions almost exactly, with MAE 0.03, RMSE 0.17, JS distance 0.0016, and Cramér's V 1.00.","keywords":["artificial surveys","LLMs","mobility preferences","synthetic data","persona-based generation","MiD 2017","Germany","survey simulation"],"falsifier":"Run the Guided Persona prompt with the MiD 2017 expected response statistics removed or corrupted, keeping all other settings fixed; if MAE, RMSE, and Cramér's V collapse toward the Structured Persona baseline, the near-perfect numbers are an artifact of reciting the prompt's provided answers, not evidence that the LLM learned demographic-response dependencies.","tokens_in":4943,"feed_emoji":"📊","tokens_out":8606,"duration_ms":81321,"temperature":0.7,"pith_summary":"The paper claims that a 'Guided Persona-based AI Survey' — an LLM prompted with personas (combinations of age, education, main activity, economic status, and household type) plus expected response statistics from Germany's MiD 2017 mobility survey — reproduces observed mobility preferences almost perfectly. Among the six compared methods, this variant achieves MAE of 0.03 percentage points, RMSE of 0.17, Jensen-Shannon distance of 0.0016, and Cramér's V of 1.00, while the naive and structured baselines land roughly an order of magnitude worse. If the claim holds, researchers could generate large, low-cost, privacy-preserving synthetic survey datasets and explore hypothetical scenarios without running new field surveys. The contribution, read sympathetically, is showing that organizing generation around a demographic conditional-probability chain and guiding the LLM with benchmark statistics yields distributionally faithful synthetic responses. The limiting caveat is explicit in Section 2: the winning method 'incorporated expected response statistics from the MiD 2017 dataset,' and the same dataset is the benchmark, so the near-perfect alignment partly reflects recitation of the prompt's numbers rather than pure generalization.","feed_headline":"LLM persona surveys match real mobility data almost exactly","feed_subtitle":"German MiD 2017 preferences reproduced with MAE 0.03 and Cramér's V 1.00 — but the benchmark answers were in the prompt.","key_machinery":"The load-bearing object is the Persona, a tuple $P(A,E,M,S,H)$ fixing an age group, education level, main activity, economic status, and household type. Each Persona's population share is built from a chain of conditional probabilities, $\\pi_P = P_A(a)\\cdot P_E(e\\mid a)\\cdot P_M(m\\mid a,e)\\cdot P_S(s\\mid a,e,m)\\cdot P_H(h\\mid a,e,m,s)$, and the LLM's per-Persona responses are reweighted by those shares to form the synthetic survey. The Guided variant additionally inserts the MiD 2017 expected response statistics into the prompt, so the model is given the target answer averages while it simulates the 15,840 Personas; the evaluated output is the weighted aggregate, and it is that prompting step that separates the near-perfect scores from the weaker baselines.","core_discovery":"On its own terms, the paper's discovery is that an LLM-based generator reaches near-perfect statistical agreement with a real national travel survey when responses are produced per Persona and the prompt carries the survey's expected response distributions. The reported metrics — MAE 0.03, RMSE 0.17, JS distance 0.0016, Cramér's V 1.00 — are interpreted as the LLM capturing complex dependencies between demographic attributes and mobility preferences, not as a reproduction of the supplied statistics. The comparison against five lower-performing baselines is used to argue that both the persona structure and the guided constraints are necessary for the fidelity.","pith_inferences":["The near-perfect metrics are likely measuring prompt fidelity, not learned demographic reasoning, since the benchmark's expected response statistics are already inside the winning prompt; withholding them would reveal what the LLM can actually infer.","A direct extension would be a held-out test: guide the model with MiD 2017 statistics for some questions, ask for other questions never summarized, and see whether distributional accuracy survives.","The persona conditional-probability chain could generate a synthetic survey without any LLM, so the paper does not isolate the LLM's added value; a sampling-only baseline over the same shares would clarify that.","Transport planners could use the guided method for counterfactual scenarios, such as changing household mixes or economic-status distributions, but those counterfactuals inherit the assumption that the LLM's response profiles, not just the provided shares, are trustworthy."],"forward_implications":["Synthetic mobility surveys can be produced at scale, as the paper generates 10,000-individual populations and 15,840 Personas from prompt-level specifications.","Because only aggregate benchmark statistics are needed, the method offers a privacy-preserving way to study population-level mobility preferences without individual records.","The same recipe — persona attributes, conditional shares, and guided prompting with expected statistics — should transfer to other transport modes, other countries, and other survey years.","The large error drop between guided and non-guided variants implies that response-level constraints, not demographic alignment alone, drive distributional fidelity.","The evaluation kit (MAE, RMSE, JS distance, entropy, Cramér's V) provides a reusable standard for judging whether any synthetic survey is statistically interchangeable with a real one."],"supporting_citations":[{"why":"Supplies the benchmark dataset, demographic distributions, and expected response statistics that define the personas and ground the evaluation.","marker":"MiD, 2017"},{"why":"The LLM-agent prompting framework the authors adapt to generate synthetic populations and survey responses.","marker":"Tzachristas, 2024"},{"why":"Earlier work on LLM synthetic replacements for human survey data; its method underlies the Naive AI-survey baseline.","marker":"Bisbee et al., 2024"},{"why":"Provides a prior approach to generating synthetic data from large language models, used as context for the survey generation methods.","marker":"Choenni et al., 2023"},{"why":"Synthetic-data best practices that motivate the generation and evaluation design.","marker":"Liu et al., 2024"},{"why":"Survey of LLM data annotation and synthesis used to support the scalability and flexibility arguments.","marker":"Tan et al., 2024"}],"fun_headline_variants":["LLM surveys need the answer key to mirror mobility data","Persona-guided LLMs match survey stats only with leaked answers","LLM fidelity to mobility survey depends on prompt hints","LLM personas need supplied distributions to hit MAE 0.03","Guided LLM surveys mirror mobility patterns only when given answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success depends on treating the insertion of the survey's expected response statistics into the LLM prompt as part of the model's learning, rather than as the answer key being handed to the model.","fun_headline_variants_meta":{"raw":{"variants":["LLM surveys need the answer key to mirror mobility data","Persona-guided LLMs match survey stats only with leaked answers","LLM fidelity to mobility survey depends on prompt hints","LLM personas need supplied distributions to hit MAE 0.03","Guided LLM surveys mirror mobility patterns only when given answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3399,"prompt_tokens":841,"completion_tokens":2558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":2473}},"tokens_in":457,"tokens_out":2558,"duration_ms":19791,"temperature":1.0,"reasoning_tokens":2473,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:08:31.399302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Guided Persona prompt with the MiD 2017 expected response statistics removed or corrupted, keeping all other settings fixed; if MAE, RMSE, and Cramér's V collapse toward the Structured Persona baseline, the near-perfect numbers are an artifact of reciting the prompt's provided answers, not evidence that the LLM learned demographic-response dependencies.","supporting_citations":[{"cited_title":"APACrefauthors \\ 2017","cited_arxiv_id":null,"evidence_quote":"Supplies the benchmark dataset, demographic distributions, and expected response statistics that define the personas and ground the evaluation."},{"cited_title":"Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs","cited_arxiv_id":"2412.13233","evidence_quote":"The LLM-agent prompting framework the authors adapt to generate synthetic populations and survey responses."},{"cited_title":", Busker, T","cited_arxiv_id":null,"evidence_quote":"Provides a prior approach to generating synthetic data from large language models, used as context for the survey generation methods."}],"review_version":1}