{"id":"f7936e68-f708-4f41-a13c-d9ae5a40eb18","arxiv_id":"2502.07068","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Fine-tuning LLMs to match country-level survey response distributions with a first-token KL-divergence loss gives modest but consistent accuracy gains over zero-shot prompting, while remaining far from reliable on unseen questions.","lead":"This paper fine-tunes LLMs to predict country-level answer distributions for survey questions, using a first-token probability loss, and tests them on unseen questions, countries, and a new survey. The fine-tuned models beat zero-shot prompting, but still fall short of accurate simulation, especially on unfamiliar questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never calibrates the zero-shot baseline; part of the fine-tuning gain may be mere output smoothing, so the 'specialization' attribution is not yet established.","rationale":"I read the paper in good faith. Its central claim is not that current models can accurately simulate surveys—the authors say plainly that even the best model is far from accurate—but that specializing via first-token KL fine-tuning substantially improves over zero-shot prompting and generalizes to unseen countries, questions, and a survey. That claim is supported by consistent gains across seven models, and the paper is appropriately cautious. The weakest link is the attribution of those gains to specialization rather than to a trivial distribution-calibration effect. The zero-shot baseline is taken as the default prompt output, with no temperature, label-smoothing, or other post-hoc calibration comparison. Because the proposed loss directly reshapes output probabilities toward smoother survey distributions, and because FT[ctrl] (inference with country labels replaced) retains much of the improvement, a substantial part of the gain could be task-level output smoothing that any calibrated classifier would also exhibit. This is not a claim that the paper is wrong; it is a missing control for the paper's main inference. The reader's ground-truth-noise concern is less load-bearing: WVS country aggregates come from samples of 1,000+ respondents, so sampling error in option proportions is about one to two percentage points, which translates to JSD changes far smaller than the 0.05-0.20 improvements reported. The GLM-4 translation issue affects only the Chinese robustness experiment, not the main result. Therefore I recommend keeping the conditional verdict, with the additional condition that the authors run the temperature-scaled zero-shot control.","tokens_in":14990,"tokens_out":19489,"duration_ms":191944,"concrete_test":"For each base model, use the validation split (C1-Q2) to select a scalar temperature T (and optionally a label-smoothing parameter) applied to the zero-shot first-token option logits, optimizing the same 1-JSD and EMD metrics used in Table 4. Apply the chosen T to every test split in Table 4 and to the Pew evaluation in Table 6, and compare against the FT results with bootstrap confidence intervals over question-country entries. If temperature-scaled zero-shot matches FT within confidence intervals on both metrics, the fine-tuning-specific gain is not established; if FT remains clearly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim is that first-token KL fine-tuning 'substantially outperforms' zero-shot classifiers (Abstract, Section 6.1), which is the evidence for the paper's central conclusion that specialization drives the gains. The zero-shot baseline is never calibrated. LLM first-token softmax distributions are often overconfident, and the reported improvements (e.g., Llama3-8B-Instruct average 1-JSD 0.613 to 0.823 in Table 4) are exactly the kind of change that a simple temperature scaling of the zero-shot logits could produce, without any specialization. The Section 6.2 control comparison (FT[ctrl] versus ZS[ctrl]) shows that a large share of the improvement persists even when country labels are randomly replaced, meaning much of the gain is a smoother, more prior-like output distribution rather than learned country-specific structure. The Section 6.4 ablation covers loss functions and option ordering, but not zero-shot output calibration. If temperature scaling (or label smoothing) of the zero-shot model reaches the fine-tuned scores on unseen countries, unseen questions, and Pew, the contribution is better described as calibration than as specialization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fine-tuning method for large language models that predicts country-level survey response distributions. The method minimizes KL divergence between the model's first-token probabilities over answer options and the human response distribution from the World Values Survey. The authors evaluate seven models from three families on held-out questions, held-out countries, and the Pew Global Attitudes Survey, reporting consistent improvements over zero-shot prompting. They also analyze country-context sensitivity through a control condition, ablate the loss function and option ordering, and include a Chinese-language robustness study. The paper is honest about remaining limitations, especially for unseen questions.","tokens_in":15252,"tokens_out":7031,"duration_ms":59251,"significance":"If the central claim is fully supported, this is a useful contribution to LLM-based survey simulation: it provides a concrete fine-tuning recipe, publicly available datasets, and an honest assessment of current limits. The breadth of models tested and the held-out Pew survey are clear strengths. However, the stronger attribution to 'specialization' rather than calibration is not yet established because the zero-shot baseline is not calibrated and no uncertainty quantification is provided. These issues are addressable, so the contribution is promising but needs revision.","major_comments":[{"comment":"All reported scores are single point estimates without standard deviations, confidence intervals, or significance tests. Since each test subset contains many country-question entries, the authors should report variability (for example, bootstrap over entries or per-country or per-question standard errors) to support the claims of 'substantially outperforms' and 'consistent across seven models.' This is particularly important for comparisons with small absolute differences, such as several FT versus FT[ctrl] rows in Table 4.","section":"Section 6.1, Table 4"},{"comment":"The sentence reporting 'a 16.7% avg. (1−JSD) drop between FT[ctrl] and FT across seven models, compared to 3.7% for ZS[ctrl] and ZS' is not reproducible from Table 4. Averaging the listed averages gives FT ≈ 0.816 versus FT[ctrl] ≈ 0.789, a relative drop of about 3.2%, and ZS ≈ 0.707 versus ZS[ctrl] ≈ 0.702, a drop of about 0.7%. Please correct these numbers and re-examine the conclusion that fine-tuned models are substantially more sensitive to country context.","section":"Section 6.2, Table 4"},{"comment":"The zero-shot baseline is never calibrated. LLM first-token distributions are typically overconfident, and the control condition FT[ctrl] already yields large gains over ZS[ctrl] without using the correct country labels (for example, Llama3-8B-Instruct: 0.599 to 0.786), so part of the improvement may be due to output smoothing rather than learned country-specific structure. To support the paper's 'specialization' attribution, add a calibrated zero-shot baseline (for example, temperature scaling or label smoothing of the zero-shot logits) or explicitly reframe the contribution as distribution calibration. This is the main correctness-risk concern for the paper's central claim.","section":"Section 6.1, Tables 4 and 11"},{"comment":"The reference distributions are treated as noise-free ground truth. The WVS aggregate percentages come with sampling error (typical country samples are around 1,000 to 1,500 respondents), and the reported improvements are on the order of 0.05 to 0.10 in 1−JSD. The authors should quantify the sensitivity of the main comparison to plausible noise in the reference distributions, for example by perturbing the targets with multinomial sampling error and recomputing the metrics.","section":"Section 3.1, Table 4"}],"minor_comments":[{"comment":"The abstract states that results hold for a completely unseen survey, but Table 6 reports Pew results for only two models, not all seven. Please qualify the claim accordingly.","section":"Abstract and Section 6.3"},{"comment":"C2 and C3 overlap (Nigeria and Morocco appear in both). Please state explicitly that the test subsets are disjoint from the training countries but not from each other, and justify the overlap.","section":"Section 3.3, Table 2"},{"comment":"The control condition replaces countries 'randomly selected from the full set of countries.' If random replacement can select the original country, the control is not a pure no-country condition. Please clarify or constrain the sampling.","section":"Section 5.2"},{"comment":"There are several typos: 'Comparsion' should be 'Comparison' in Appendix headings, 'V olkswagenStiftung' in the acknowledgments should be 'VolkswagenStiftung,' and 'further revealing' in Section 6.1 should be 'further reveal.'","section":"Appendix F and Section 6.1"},{"comment":"The description of the blue-shaded area is unclear; please define the lower and upper boundaries more precisely.","section":"Figure 3 caption"},{"comment":"In Table 7, please clarify that the 'Shuffled' row evaluates a model on a test set with shuffled option order, not a shuffled training order.","section":"Section 6.4, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The missing calibration baseline is the key issue. If the authors can show that a simple temperature-scaling or label-smoothing baseline on zero-shot logits does not reproduce the fine-tuning gains, I would be inclined to accept after revision. The numerical inconsistency in Section 6.2 should be checked carefully, as it may indicate a miscalculation rather than a typo."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper with a real result, but the main selling point—country-specific specialization—is only partially supported because the zero-shot baseline is never calibrated.\n\nThe new thing here is the task: predicting country-level response distributions for survey questions, and a method that fine-tunes LLMs on first-token probabilities with KL divergence. The evaluation is genuinely held out: unseen questions, unseen countries, and a completely unseen survey (Pew). The gains over zero-shot are consistent across seven models from three families, and the code and data are released. The paper is also honest about how far the best models are from accurate simulation, especially on unseen questions. That is a useful, reproducible empirical contribution.\n\nThe soft spots are real but not fatal. The stress-test note is right: there is no calibrated zero-shot baseline. The paper's own control, FT[ctrl], which is fine-tuned with randomly replaced country labels, jumps from ZS[ctrl] at 0.599 to FT[ctrl] at 0.786 for Llama3-Instruct, while the added gain from true country labels is only to 0.823. So most of the improvement comes from the model learning the marginal response distribution and producing smoother first-token probabilities, not from country-specific structure. A simple temperature scaling or label-smoothing baseline for zero-shot would be needed to support the claim that this is specialization in the sense the title suggests. The paper's RQ2 discussion is in the right direction, but it does not include that baseline.\n\nThe other issues are more minor. All results are point estimates with no standard deviations or significance tests. Given the reference distributions are themselves survey aggregates with sampling error, the authors should at least report the margin of error or do a bootstrap. The Chinese robustness experiment depends partly on GLM-4 translations (footnote 3), which weakens that particular claim but not the English-language core.\n\nOverall, the central empirical claim—first-token KL fine-tuning improves distribution simulation—is solid and worth publishing. The attribution to country-specific specialization is overstated relative to the evidence, but that is fixable with the control experiments above. This paper deserves a serious referee: it ships code and data, asks a meaningful question, and evaluates it in a mostly sound way. I'd recommend sending it to review, with the calibration baseline and error bars as required revisions.","headline":"Useful empirical paper on fine-tuning LLMs to predict survey response distributions, but the main attribution to country-specific specialization is undercut by the missing zero-shot calibration baseline.","tokens_in":15737,"tokens_out":3066,"would_cite":false,"duration_ms":27361,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning LLMs on first-token probabilities can substantially improve simulation of country-level survey response distributions, even for unseen countries, questions, and surveys.","keywords":["survey simulation","large language models","fine-tuning","first-token probability","KL divergence","World Values Survey","Pew Global Attitudes Survey","response distributions"],"falsifier":"Compute country-level response distributions from the WVS microdata using respondent weights and survey design, then re-run the fine-tuned models against those weighted references; if the 1-JSD gaps between fine-tuned and zero-shot models (roughly 0.05 to 0.10 points) fall inside the surveys' margins of error, the claim of substantial improvement would not survive.","tokens_in":14829,"feed_emoji":"🗳️","tokens_out":2509,"duration_ms":23120,"temperature":0.7,"pith_summary":"This paper asks whether large language models can be specialized, rather than merely prompted, to predict how whole populations would answer survey questions. The authors propose a fine-tuning method that aligns a model's first-token probability distribution over answer options with the observed country-level response distribution from the World Values Survey, using a Kullback-Leibler divergence loss. They show that this method substantially outperforms zero-shot prompting and other baselines across seven models and three model families, and that the gains carry over to unseen countries, unseen questions, and a completely unseen survey (the Pew Global Attitudes Survey). The paper also reports that even the best fine-tuned models remain far from accurate, especially on unseen questions, and that model predictions are consistently less diverse across countries than the human data. If the method proves robust, it would offer a cheaper, faster way to approximate group-level survey results for social science and policy before investing in expensive data collection.","feed_headline":"Fine-tuned LLMs predict survey answers across countries","feed_subtitle":"Aligning the first-token distribution to human responses beats zero-shot prompting, even on unseen questions and surveys.","key_machinery":"The central mechanism is first-token probability alignment: the model's input is formatted so that the first output token must be one of the valid answer options, the logits for those options are normalized with a softmax to form a predicted distribution, and the training loss is the KL divergence between that predicted distribution and the human response distribution for the given country and question. This turns distribution simulation into a calibration task rather than a single-answer prediction task, and it is what allows a decoder-only LLM to output a full response distribution from just the first token. The format field that restricts the first-token vocabulary, together with LoRA for efficient fine-tuning, carries most of the method's effectiveness.","core_discovery":"The central claim is that specializing LLMs through first-token probability alignment fine-tuning substantially improves the simulation of country-level survey response distributions, and that these gains generalize beyond the training distribution. The authors devise a training objective that minimizes the KL divergence between the softmax distribution over the first-token logits of the valid answer options and the human response distribution for each country-question pair, using LoRA for parameter-efficient tuning. Across seven models (Vicuna1.5-7B/13B, Llama3-8B-Base/Instruct, and Distil-Qwen-7B/14B/32B), the fine-tuned models consistently beat zero-shot prompting and control baselines (shuffled countries, KNN, average-culture, and JSON prompting) on both 1-JSD and EMD metrics. The improvement holds for unseen WVS questions and countries, for the Chinese version of the WVS, and for the held-out Pew survey. At the same time, the paper's own results show systematic limitations: unseen questions remain the hardest setting, and all models, fine-tuned or not, produce less cross-country diversity than humans do.","pith_inferences":["The method could plausibly be extended beyond country-level aggregates to subnational groups, demographic segments, or even individual-level response distributions, since the same first-token alignment loss applies to any target distribution.","The persistent under-diversity across countries suggests a structural limit of likelihood-based fine-tuning: the model learns an average cultural prior and may require explicit diversity-promoting terms to match inter-country variance.","A natural testable extension is to compare first-token KL fine-tuning against fine-tuning that predicts the full response distribution in one step (e.g., via a softmax over all options decoded at once); the paper's ablation with JSON-style prompting only tests the zero-shot variant, leaving a fine-tuned multi-token baseline unexplored."],"forward_implications":["If the method is correct, LLMs specialized this way could provide low-cost, preliminary estimates of country-level opinions on new questions without fielding a survey.","The consistent improvement across base and instruct models of different families suggests that first-token KL fine-tuning is a robust specialization recipe, not a quirk of one model architecture.","Because gains transfer to a completely unseen survey, the approach may generalize beyond the World Values Survey to other global questionnaires with the same multiple-choice format.","The paper's finding that even the best models are weak on unseen questions and under-diverse across countries implies that such simulations should not yet replace real surveys for consequential decisions."],"supporting_citations":[{"why":"Provides the World Values Survey wave 7 cross-national dataset that is the main training and evaluation ground truth.","marker":"Haerpfer et al. (2022)"},{"why":"Supplies the LoRA parameter-efficient fine-tuning method used to train all models.","marker":"Hu et al. (2022)"},{"why":"Supplies the GlobalOpinionQA template and the 1-JSD evaluation metric, including the Pew dataset used as the unseen-survey test.","marker":"Durmus et al. (2023)"},{"why":"Defines the Earth Mover Distance metric used as the second distribution-alignment evaluation.","marker":"Rubner et al. (1998)"},{"why":"Establishes the prior approach of prompting LLMs to simulate human samples, the baseline this work extends and outperforms.","marker":"Argyle et al. (2023)"},{"why":"Provides the Vicuna1.5 models, one of the three model families tested.","marker":"Chiang et al. (2023)"},{"why":"Provides the Llama3-8B Base and Instruct models, another central model family.","marker":"AI@Meta (2024)"},{"why":"Provides the DeepSeek-R1-Distill-Qwen models used as the third model family.","marker":"Guo et al. (2025)"},{"why":"Documents the A-bias in option ordering that motivates the paper's shuffled-options robustness check.","marker":"Dominguez-Olmedo et al. (2023)"}],"fun_headline_variants":["Fine-tuned LLMs align to human survey distributions","First-token fine-tuning outperforms zero-shot for survey simulation","Specializing LLMs to simulate global survey response distributions","Aligning first-token logits beats prompting on unseen surveys"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper treats the published country-level percentages from the WVS and Pew surveys as exact, noise-free human response distributions, and it uses machine-translated text for part of the Chinese training data, so any errors in those references would shrink the reported improvements.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned LLMs align to human survey distributions","First-token fine-tuning outperforms zero-shot for survey simulation","Specializing LLMs to simulate global survey response distributions","Aligning first-token logits beats prompting on unseen surveys"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000833,"raw_usage":{"total_tokens":3636,"prompt_tokens":947,"completion_tokens":2689,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":2622}},"tokens_in":563,"tokens_out":2689,"duration_ms":17873,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:53:51.239085+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute country-level response distributions from the WVS microdata using respondent weights and survey design, then re-run the fine-tuned models against those weighted references; if the 1-JSD gaps between fine-tuned and zero-shot models (roughly 0.05 to 0.10 points) fall inside the surveys' margins of error, the claim of substantial improvement would not survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Earth Mover Distance metric used as the second distribution-alignment evaluation."},{"cited_title":"Gonzalez, Ion Stoica, and Eric P","cited_arxiv_id":null,"evidence_quote":"Provides the Vicuna1.5 models, one of the three model families tested."}],"review_version":1}