{"id":"3fc40270-5da5-4de1-9cd0-39361d9bbe61","arxiv_id":"2505.04196","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning a lightweight LLM with attribute sequences ordered by a Bayesian network produces synthetic populations with about 95% feasibility while preserving recall comparable to deep generative models.","lead":"This paper fine-tunes a small open-source language model to generate synthetic residents for travel-demand simulations, using a Bayesian network to decide the order in which attributes are written out. The method reportedly yields about 95 percent feasible attribute combinations, compared with roughly 80 percent for deep generative baselines, while keeping diversity roughly the same.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95% feasibility figure is measured against a proxy h-population whose representativeness is assumed, not tested; if the merged HTS contains sampling zeros or survey artifacts, the headline number does not transfer to real populations.","rationale":"The paper is transparent about its semi-synthetic evaluation and provides open source code, which is genuine support for internal reproducibility. The central claim, however, is an empirical statement about real-world feasibility and diversity, and the only evidence is computed against the merged HTS treated as the population. This is exactly the assumption identified by the reader as weakest, and it is load-bearing: every reported precision and recall number in Table 3 is a membership test against h-population cells, so any mismatch between the h-population and the true population changes both the numerator and the denominator of the headline metrics. The paper's own statement that sampling zeros are minimal is an assertion, not a demonstrated property of the merged data. The random-ordering baseline also deserves attention because it suggests the BN ordering may not be the active ingredient, but that concern primarily affects the novelty of the method rather than the external validity of the feasibility estimate. With the proxy concern unresolved, the appropriate outcome is to retain the conditional verdict: accept only after an independent/held-out validation that does not rely solely on the same merged h-population. I therefore recommend no change to the reader's verdict.","tokens_in":19266,"tokens_out":8481,"duration_ms":95759,"concrete_test":"Hold out one HTS wave entirely: train the LLM-BN pipeline only on 2010/2016 h-samples and evaluate precision/recall against the 2021 wave as an independent h-population (or the reverse). If LLM-BN precision on the held-out wave drops materially below the reported 95.3% or its F1 advantage over DGM-WGAN disappears, the headline result depends on the merged-proxy assumption and should be re-qualified. Repeating the check with several random h-sample draws and reporting the mean/range would also determine whether the single reported configuration is stable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines the h-population as the merged Korean HTS (N=1,006,391), draws a 5% h-sample from it, trains on the h-sample, and evaluates precision/recall (Eq. 9) against the h-population. The paper explicitly states: \"we assume that the presence of sampling zeros in our data is minimal.\" The central claim, 95.3% feasibility for LLM-BN in Table 3, is therefore conditional on the h-population being a faithful stand-in for the true population and on structural zeros in it being negligible. If the merged survey waves miss real subpopulations or contain coding artifacts, then (i) genuinely feasible generated combinations that are absent from the h-population are misclassified as structural zeros, lowering measured precision, while (ii) implausible combinations present in the survey artifact are counted as feasible, inflating precision. The same h-population is also used to choose the temperature and epoch settings via F1-based calibration (Section 5.2, Figure 7), so the reported number is the best proxy-selected configuration rather than an independent estimate. A secondary internal-validity concern is that LLM-Random achieves F1 85.2 versus LLM-BN 84.6, so the BN-ordering component is not shown to improve overall quality even under the proxy assumption; however, the more load-bearing issue is that no external or held-out ground truth validates the headline feasibility figure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-BN, a method that fine-tunes a lightweight open-source LLM (distilled GPT-2) on a 5% sample from a one-million-record Korean household travel survey, using a Bayesian-network-derived topological attribute ordering to serialize records during training. Generation is controlled by decoding temperature and fine-tuning depth. Evaluated against the full survey as a hypothetical population, the method reports precision 95.3%, recall 76.0%, and F1 84.6%, compared with DGM-WGAN at 81.4%, 80.8%, and 81.1%, and claims approximately 95% feasibility with comparable diversity. The paper also benchmarks GPT-4o with few-shot prompting, an LLM fine-tuned with random attribute orderings, two DGMs, a prototypical baseline, and reports computational and API-cost estimates.","tokens_in":19552,"tokens_out":5790,"duration_ms":58655,"significance":"If the empirical results are robust, the contribution is practically significant: the method achieves high feasibility on a large-scale synthetic-population task using an 82-million-parameter open-source model that can be fine-tuned and run on a standard GPU, with source code and data made available. The paper also includes a useful sensitivity analysis of temperature and epochs and explicit cost comparisons. The main strengths are the clear statement of the proxy-population assumption, the reproducible experimental setup in principle, and the lightweight deployment argument. However, several internal-validity issues currently prevent me from treating the headline feasibility figure as established.","major_comments":[{"comment":"The claim that the BN ordering improves overall quality is not supported by the reported numbers: LLM-Random achieves F1 85.2 versus LLM-BN 84.6, and recall 80.3% versus 76.0%, with LLM-BN higher only on precision (95.3% versus 90.8%). Since LLM-Random differs from LLM-BN only in the attribute ordering, the proposed BN component is not shown to be beneficial on the primary F1 criterion. Please report multiple seeds or paired comparisons and state explicitly under which objective the BN ordering is preferable.","section":"§5.2, Table 3"},{"comment":"The h-population proxy assumption is load-bearing for the central feasibility claim. Precision and recall in Eq. (9) are computed against the merged Korean HTS, and Section 3 explicitly states 'we assume that the presence of sampling zeros in our data is minimal.' If the merged survey misses real subpopulations or contains coding artifacts, then feasible generated combinations absent from the survey are counted as structural zeros, lowering precision, while artifact combinations present in the survey are counted as feasible, inflating precision. The paper should add external validation (for example, comparison with census marginals), sensitivity analysis with respect to sample size, or at minimum an explicit limitations subsection explaining how the numbers transfer to a real population.","section":"§3, §5.2"},{"comment":"The hyperparameters are calibrated on the same evaluation metric and evaluation population used to report the final results. Section 5.2 states 'we calibrate the hyperparameters based on overall quality, F1 score,' and Figure 7 shows the F1-based selection of temperature and epochs against the h-population. The Table 3 numbers are therefore the best proxy-selected configuration rather than an independent estimate. A validation split or a nested model-selection procedure is needed to avoid selection bias in the headline feasibility figure.","section":"§5.2, §5.3, Figure 7"},{"comment":"The LLM-BN result is not numerically reproducible from the manuscript as written. Table 3 reports 120,541 unique combinations, recall 76.0%, precision 95.3%, and F1 84.6% for LLM-BN, while Table 4 reports, for the same Distilled GPT-2 model under the LLM-BN method, 128,950 combinations, recall 76.7%, precision 94.5%, and F1 84.7%. The text says the same temperature and epochs were applied across model sizes, but does not say whether these settings match Table 3. This discrepancy suggests seed or configuration sensitivity, and without error bars or a precise configuration statement the reader cannot tell which number is the headline result.","section":"Table 4 versus Table 3"}],"minor_comments":[{"comment":"The header 'SMRSE' appears to be a typo for 'SRMSE' as used in Section 4.4.1.","section":"Table 3"},{"comment":"Equation numbering jumps from Eq. (4) to Eq. (6); the missing Eq. (5) should be renumbered or removed.","section":"§4.2"},{"comment":"The first column header contains the typo 'DGN-VAE'; it should read 'DGM-VAE' to match the benchmark names elsewhere.","section":"Table 2"},{"comment":"The fine-tuning cost columns are confusing: the column 'Train (per 1M tokens)' followed by '40 Epochs for h-sample' appears to report total costs (e.g., $5,011.70), not per-million-token rates; please clarify whether these are per-token prices or total amounts.","section":"Table 6"},{"comment":"The table caption should state explicitly which hyperparameter configuration is used and whether it is the same as the configuration that produced Table 3; currently the reader cannot determine this.","section":"§5.4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation framework, including the precision/recall definitions and the h-population/h-sample split, comes from Kim and Bansal (2023), whose author E.J. Kim is a coauthor of this manuscript. This is not inherently problematic, but the overlap should be acknowledged in the cover letter, and the novelty of the present evaluation relative to that prior work should be made explicit. No other concerns about citation or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuine, reproducible empirical comparison of fine-tuned LLMs for population synthesis, but the headline feasibility number is measured against a proxy population, and the proposed BN ordering does not beat random ordering on F1.\n\nWhat's new: applying GReaT-style LLM fine-tuning to population synthesis with a BN-derived topological ordering as an inductive bias. That specific combination is new. The paper also includes a sensible comparison set: DGMs, proprietary GPT-4o few-shot, and LLM-Random as the GReaT-style control. Code and data are public, which is real credit. The cost analysis for lightweight local models vs proprietary APIs is practically useful, and the authors are transparent that their evaluation treats the merged HTS as a hypothetical population.\n\nSoft spots: The biggest is that every precision/recall number is conditional on the h-population being a faithful stand-in for the true population. The authors state they assume sampling zeros are minimal; if the HTS misses real subpopulations or has artifacts, the 95% feasibility claim doesn't transfer. Second, hyperparameters (temperature, epochs) are calibrated on F1 against the same h-population used for evaluation, so the headline number is selection-best, not an independent estimate. Third, there are no error bars or multiple seeds; for a method with as much randomness as LLM decoding, that's a real gap. Fourth, and most important: LLM-Random achieves F1 85.2 vs LLM-BN 84.6. The paper does not show that BN ordering improves overall quality; it only trades recall for precision. The paper frames this as a controllable trade-off, which is honest, but the title and abstract imply the BN component is a win. Finally, minor: the text in Section 5.5 says 2.67 hours for inference while Table 5 says 3.84; also LLM methods have much higher SRMSE than DGMs, which the paper dismisses too quickly.\n\nVerdict: The central empirical claim about DGMs vs fine-tuned LLMs is credible within the proxy evaluation. The marginal value of BN ordering is unproven. This paper is for transportation ABM researchers and anyone working on LLM-based tabular generation. It deserves a serious referee and likely conditional acceptance after the authors add a proper validation split, multiple seeds, and a cleaner comparison of BN vs random ordering.","headline":"A reproducible applied study of LLM-based population synthesis with a credible feasibility gain against DGMs, but the BN ordering is not shown to beat random ordering and all headline numbers are proxy-dependent.","tokens_in":20099,"tokens_out":2777,"would_cite":false,"duration_ms":25892,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small fine-tuned language model generates realistic synthetic city populations at 95 percent feasibility.","keywords":["population synthesis","activity-based models","large language models","Bayesian network","feasibility","diversity","zero-cell problem","fine-tuning"],"falsifier":"Draw a 5% sample from an independent full-coverage census or population register, fine-tune the same model on it, and check whether the generated combinations appear in the full register at the claimed roughly 95% rate; if precision falls well below that, the proxy population, not the model, is carrying the feasibility result.","tokens_in":19058,"feed_emoji":"🏙️","tokens_out":6123,"duration_ms":53663,"temperature":0.7,"pith_summary":"The paper proposes a way to synthesize populations of artificial residents for activity-based travel-demand models: fine-tune a lightweight open-source language model on household travel survey records, formatting each person as a text sentence and ordering the attributes according to a Bayesian network learned from the data. The claim is that this ordering disciplines the autoregressive generation so that about 95 percent of generated attribute combinations are combinations that actually occur in the population, while the model still recovers rare but plausible combinations that are missing from the small training sample. The result matters because downstream activity scheduling in agent-based models is only as good as the synthetic population it starts from; implausible combinations propagate as errors through the simulation. The paper also shows the method runs on a standard personal computer, whereas proprietary few-shot prompting is both less diverse and much more expensive.","feed_headline":"Fine-tuned LLM builds synthetic populations that are 95 percent feasible","feed_subtitle":"Small open-source model beats deep generative nets on feasibility while keeping diversity, at a fraction of the cost.","key_machinery":"The load-bearing object is the topological ordering of attributes. A Bayesian network is first learned from the training sample with a score-based search restricted to at most one parent per node, producing a chain-like directed acyclic graph; that graph yields an ordering in which each attribute is written only after its structural predecessors. During fine-tuning, each record is converted into a natural-language sentence following that order, and during generation the model is prompted with the first attribute's text and samples the rest autoregressively. Together with decoding temperature and the number of fine-tuning epochs, the ordering is what lets the model separate plausible but unobserved combinations from implausible ones.","core_discovery":"On the paper's own terms, the central discovery is that a distilled GPT-2 model (about 82 million parameters) fine-tuned on a 5% sample of a one-million-record merged household travel survey, with attributes serialized in a topological order from a learned Bayesian network, reaches 95.3% precision and 76.0% recall (F1 84.6%) against the full survey taken as a hypothetical population. This beats the Wasserstein GAN baseline (81.4% precision, 80.8% recall, F1 81.1%) and the few-shot proprietary LLM baseline (84.6% precision but only 50.2% recall, F1 63.0%). The interpretation offered is that the language model contributes semantic knowledge about which attribute combinations are plausible, while the Bayesian-network ordering constrains the autoregressive search space, and the fine-tuning depth plus decoding temperature tune the precision-recall trade-off.","pith_inferences":["If the Bayesian-network ordering is the main mechanism, then swapping in a causal graph or a knowledge graph built from domain rules should produce similar or better feasibility on populations with different demographic structures; the paper does not test this.","The semantic priors come from the language model's pretraining corpus, which is largely English and Western; a direct test would apply the same pipeline to a non-Korean population and watch whether precision holds.","The model-size insensitivity suggests that with enough fine-tuning epochs the task reduces to structured recall; a testable extension is whether larger models become necessary when the attribute space grows beyond 13 attributes or when attributes interact non-locally.","The learned dependency order from one city could be transferred to another city to test whether the demographic structure is transportable, which would make the method cheaper to deploy in new regions."],"forward_implications":["Synthetic populations for activity-based models can be produced on laptop-class hardware, removing the need for proprietary APIs or large GPU clusters.","The feasibility-diversity trade-off becomes a controllable dial: the same model can be tuned for high precision or higher recall depending on the downstream application.","A smaller language model (82 million parameters) matches or beats much larger ones on this task, suggesting the bottleneck is structure, not raw model scale.","Better population inputs should reduce downstream error propagation in activity schedule simulation, since implausible attribute combinations no longer enter the pipeline.","The method recovers attribute combinations missing from the 5% training sample (which covers only 56.4% of the population's combinations) and reaches 76% recall, so rare but plausible population segments are not dropped."],"supporting_citations":[{"why":"Defines the feasibility/diversity metrics and the h-population validation strategy that the paper's central comparison and numbers rely on; also provides the deep generative model baselines.","marker":"Kim and Bansal, 2023"},{"why":"Introduces LLM fine-tuning on text-formatted tabular rows with random attribute orders, the direct predecessor that LLM-BN modifies by using a Bayesian-network ordering.","marker":"Borisov et al., 2023"},{"why":"Establishes the Bayesian network approach to population synthesis that supplies the method's attribute-dependency structure.","marker":"Sun and Erath, 2015"},{"why":"Brings deep generative modeling into population synthesis and defines the VAE-style baseline and zero-cell problem framing.","marker":"Borysov et al., 2019"},{"why":"Provides the K2 score used in the score-based Bayesian network structure learning.","marker":"Cooper and Herskovits, 1992"},{"why":"Supplies the distilled GPT-2 backbone that the proposed method fine-tunes.","marker":"Sanh et al., 2019"},{"why":"Source of the hill-climb search algorithm used to learn the DAG and the ordering.","marker":"Tsamardinos et al., 2006"},{"why":"Basis for the few-shot proprietary LLM baseline and its prompt-engineering setup, which the paper shows underperforms.","marker":"Liu et al., 2024a"}],"fun_headline_variants":["LLM + Bayesian net hits 95% feasibility in population synthesis","Lightweight LLM achieves 95% feasible synthetic populations","Open-source LLM synthesizes populations with 95% feasibility","Small LLM beats GANs on population feasibility, keeps diversity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the merged one-million-record Korean household travel survey as a stand-in for the true population and assumes that attribute combinations missing from it are negligible, so every feasibility number is measured against that proxy.","fun_headline_variants_meta":{"raw":{"variants":["LLM + Bayesian net hits 95% feasibility in population synthesis","Lightweight LLM achieves 95% feasible synthetic populations","Open-source LLM synthesizes populations with 95% feasibility","Small LLM beats GANs on population feasibility, keeps diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1536,"prompt_tokens":1001,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":463}},"tokens_in":617,"tokens_out":535,"duration_ms":5068,"temperature":1.0,"reasoning_tokens":463,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:34:24.755335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Draw a 5% sample from an independent full-coverage census or population register, fine-tune the same model on it, and check whether the generated combinations appear in the full register at the claimed roughly 95% rate; if precision falls well below that, the proxy population, not the model, is carrying the feasibility result.","supporting_citations":[{"cited_title":"In contrast, we use a dataset constructed by merging the South Korean HTS conducted in 2010, 2016, and 2021, resulting in over one million records","cited_arxiv_id":null,"evidence_quote":"Establishes the Bayesian network approach to population synthesis that supplies the method's attribute-dependency structure."}],"review_version":1}