{"id":"d38f494f-c2f2-40ce-b98a-a6ba82cc2459","arxiv_id":"2608.06115","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Anacreon claims state-of-the-art individual-level survey prediction by clustering public text and training a separate adapter per cluster, but its evaluation is on self-mined questions and lacks an external benchmark.","lead":"Anacreon is an AI system that splits public text into clusters of similar people and fine-tunes a separate model for each cluster to predict survey answers. The authors report a record accuracy of 0.775, but the test questions were mined from the same text used for training, so the claim is not independently verified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim (0.775) is undermined by a circular evaluation: questions are mined from the training corpus and the same held-out split is used for both checkpoint selection and final scoring, so the number is not an unbiased estimate.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern that I found: the evaluation is not externally sourced, and the same held-out data is used for both checkpoint selection and final scoring. This is not a stylistic issue; it directly determines whether the paper's central claim (state-of-the-art ordinal alignment of 0.775) is credible. The paper's own text provides the evidence: Section 3.2 says questions are mined from the text; Section 3.3 uses 'held-out data' in Eq. 5 for checkpoint selection; Section 4.2 reports results on that same held-out split. There is no separate validation set and no external benchmark. The abstract's 'externally sourced survey' is contradicted by the method. This is a clear case of selection bias and potential data contamination, and it is enough to reject the paper's central claim as unsupported. The concrete test I propose—using an external survey and a disjoint validation split—would settle whether the 0.775 figure is genuine or inflated. I agree with the reader's recommendation to reject, so the verdict remains unchanged.","tokens_in":11390,"tokens_out":3465,"duration_ms":26781,"concrete_test":"Use a genuinely external, pre-existing survey (e.g., Twin-2K-500 or the mega-study's held-out items) as the test set, with no mined questions. For checkpoint selection, hold out a separate temporal slice (e.g., 10% from the middle of each cluster) and use the newest 10% only as the final test set. Report ordinal alignment from this protocol. If the number falls below the prior SOTA (0.748) or even below the original 0.775, the circular evaluation is confirmed. Also run a control where clusters are shuffled randomly before splitting, to test whether the result depends on the authorship embedding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 4.2 assert state-of-the-art ordinal alignment of 0.775. This claim is load-bearing: if true, per-cluster fine-tuning on mined public text beats prior digital-twins. But the evaluation protocol does not support an unbiased estimate. Section 3.2 states that 'Survey questions are also mined from the text,' directly contradicting the abstract's 'externally sourced survey.' Because the same corpus supplies both training records and test questions, a model can achieve high ordinal alignment by memorizing or pattern-matching the source text rather than by generalizing to new questions. More critically, Section 3.3 selects the best checkpoint by maximizing Eq. (5) on 'held-out data,' and Section 4.2 then reports results on the same temporal 20% split. This is selection on the test set: the reported 0.775 is the maximum over checkpoints of a score that includes Aord, so it is optimistically biased. The paper does not include a disjoint validation split or an external benchmark. Table 1 itself concedes that comparisons are 'indicative rather than a controlled comparison.' Therefore the central SOTA claim is unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Anacreon, an LLM-based audience simulator that clusters a public text corpus around seed individuals, trains a per-cluster LoRA adapter on top of a Gemma 4 12B base, and augments records with mined attributes and a chain-of-emotion. The central empirical claim is a state-of-the-art ordinal alignment of 0.775 on what the abstract calls a large, externally sourced survey, together with a top-1 accuracy of 0.679 on mined multiple-choice questions. The method is presented with attention to known failure modes such as prompt brittleness and positivity bias. However, the evaluation has two load-bearing problems: the survey questions are not externally sourced but mined from the same corpus used for training, and the checkpoint-selection procedure in Eq. (5) uses the same held-out split on which the final scores are then reported. As a result, the headline 0.775 figure is not an unbiased estimate of generalization to new questions, and the state-of-the-art claim is not supported by the evidence presented.","tokens_in":11683,"tokens_out":4280,"duration_ms":37332,"significance":"If the claimed result were established by a valid, externally grounded evaluation, it would be a meaningful advance for individual-level human simulation: prior digital-twin systems report ordinal alignment values around 0.717-0.748, so a validated 0.775 would be a clear improvement. The paper also deserves credit for addressing two documented failure modes in LLM simulation (option-order prompt brittleness and positive/sycophantic bias) and for explicitly acknowledging in Table 1 that cross-dataset comparisons are indicative rather than controlled. These strengths do not compensate for the evaluation-circularity problem, because the paper's main contribution is the empirical claim itself. The method may be interesting, but the manuscript as written does not establish that it outperforms prior systems.","major_comments":[{"comment":"The abstract and §4.2 describe the evaluation as a 'large, externally sourced survey,' but §3.2 states that 'Survey questions are also mined from the text.' The test questions therefore come from the same public corpus that supplies the training records. Even though survey questions are mined separately on train and test splits, the ground truth is not an external survey instrument; it is derived from the same population and the same text genre used for training. This directly contradicts the abstract and means the 0.775 figure should not be presented as performance on a new, externally administered survey.","section":"Abstract; §3.2"},{"comment":"The best checkpoint is chosen by maximizing α·A_mc(θ) + β·A_ord(θ) - γ·PPL(θ) on 'held-out data,' and §4.2 then reports the median per-cluster A_ord on what appears to be the same temporal 20% split described in §3.1. No disjoint validation split is described anywhere in the paper. Selecting the checkpoint with a criterion that includes A_ord and then reporting A_ord on the same data makes the reported 0.775 a selected maximum rather than an unbiased estimate of performance; the magnitude of the optimistic bias is unknown and could be substantial, especially with 420 clusters and 400 retained models.","section":"§3.3, Eq. (5); §4.2"},{"comment":"The 'state of the art' claim rests on comparisons across different datasets, different survey instruments, and somewhat different experimental protocols. The table caption itself concedes that the scores are 'indicative rather than a controlled comparison.' A higher number on a non-identical evaluation does not by itself establish state-of-the-art status. A controlled comparison on a common benchmark or a matched external survey is required before the headline claim can be made.","section":"Table 1; §4.2"}],"minor_comments":[{"comment":"The sentence 'Due to the positivity bias in LLM responses [10]. most of the ordinal questions...' contains a misplaced period after the citation; it should read 'responses [10], most of...'.","section":"§3.2"},{"comment":"The paper does not report the number of seed individuals, the minimum data threshold for seed inclusion, the clustering algorithm, or the selection weights α, β, and γ in Eq. (5). These details are needed for reproducibility.","section":"§3.3"},{"comment":"The pooled signed-error analysis reports that each model was prompted five times with shuffled options, but the aggregation of those five responses into a single prediction is not described; the reader cannot tell whether the reported error uses the mean, median, or a single sample.","section":"§4.1"},{"comment":"Figure 7 is labeled 'illustrative' and appears to be a schematic representation rather than measured data; the text should state explicitly that the comparison is not an empirical measurement of Anacreon's skew against human test-retest.","section":"Figure 7; §5"},{"comment":"The discussion refers to 'a belief network [27]' as an analogy for the cluster training data, but the connection is not developed; as written it is a metaphor rather than a formal or quantitative claim.","section":"§5"}],"recommendation":"reject","confidential_remarks":"The contradiction between the abstract's 'externally sourced survey' and §3.2's 'survey questions are also mined from the text' is not a wording issue: the headline result depends on the independence of the test data. Combined with checkpoint selection on the same held-out split, the evaluation protocol invalidates the paper's central claim as presented. I would encourage the editor to consider whether the authors can supply a genuinely external survey or a properly separated validation set; without that, the manuscript's empirical contribution is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the architecture is real: a mixture of per-cluster LoRA adapters on a 12B base, with authorship embeddings used to cluster a corpus, is a sensible way to attack individual-level heterogeneity. Second, the headline number, 0.775 ordinal alignment, is not an unbiased estimate. The paper itself gives you the evidence for that. Section 3.2 says \"Survey questions are also mined from the text,\" which directly contradicts the abstract's \"externally sourced survey.\" Section 3.3 selects the best checkpoint by maximizing Eq. (5) on the same held-out temporal split that Section 4.2 then uses to report the final score. That is selection on the test set. Table 1 concedes the comparisons are \"indicative rather than a controlled comparison,\" so the SOTA claim is unsupported.\n\nWhat the paper does well: it is clearly written, the motivation is grounded in a real literature (Graunt to Simulmatics to digital twins), and it tackles known failure modes—prompt brittleness and positive bias—with concrete techniques. The chain-of-emotion is a reasonable extension of prior appraisal-based work, and the discussion of variance versus human test-retest is honest. The engineering is substantial: 420 clusters, 110B parameters, and a deployed B2B system. Credit is due there.\n\nWhere it falls short is the evaluation. Beyond the circularity, the comparison set is not controlled. Park et al. is interview-grounded and tested on different questions; Twin-2K-500 and the mega-study use different instruments. So the 0.775 number is not comparable to 0.748 without a shared benchmark. The paper needs a disjoint validation split and an external benchmark, ideally on a standard dataset like a published survey, to make the SOTA claim credible.\n\nWho is this for? Researchers working on LLM-based human simulation, particularly digital twins and audience modeling. The architecture idea is worth exploring, but the current paper does not provide a valid test of it. I would send it to peer review because the idea is meaningful and the flaws are fixable—but it needs major revision. The claims about state of the art should be withdrawn until the evaluation is clean.\n\nRecommendation: send to a serious venue, require the authors to fix the evaluation, and do not accept the current version.","headline":"A well-written paper with a genuinely new architecture, but its central SOTA claim rests on a circular evaluation: test questions are mined from the training corpus, and the same held-out split is used for both checkpoint selection and final scoring.","tokens_in":12189,"tokens_out":2637,"would_cite":false,"duration_ms":20019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Anacreon claims that a mixture of per-cluster LLM adapters trained on public text reaches state-of-the-art individual-level survey accuracy, with ordinal alignment 0.775, on a narrow audience-simulation domain.","keywords":["human simulation","LLM simulators","ordinal alignment","digital twins","mixture of minds","chain-of-emotion","contrastive authorship embedding","survey prediction"],"falsifier":"Ask Anacreon to answer a newly written survey on the same SME-merchant domain whose questions were not mined from the training corpus, and compare its ordinal alignment on those fresh items with the reported 0.775; a large drop would show the score reflects question-format familiarity rather than generalizable individual prediction. A quicker check is to recompute the score after excluding the newest-20% held-out records from the checkpoint-selection objective in eq. (5).","tokens_in":11162,"feed_emoji":"🎯","tokens_out":10774,"duration_ms":81551,"temperature":0.7,"pith_summary":"The paper argues that LLM-based human simulators capture a population's average opinions while flattening its variation, and that the way out is to specialise: model a narrow, well-defined audience with a separate trained model per cluster of similar people. Anacreon does this by learning a contrastive authorship embedding that separates individuals in public text, clustering the corpus around seed people, then fine-tuning a dedicated LoRA adapter—a 'mixture of minds'—on a Gemma 4 12B base for every cluster. Records are augmented with harvested demographics, psychological traits, and mined survey answers, and with a chain-of-emotion that traces emotional state from stimulus to response; training also shuffles response options and balances positive and negative framings to counter prompt brittleness and sycophancy. On a large externally sourced survey of SME merchants, Anacreon reports ordinal alignment 0.775, which the paper presents as the best individual-level accuracy among systems that report the measure, ahead of 0.717, 0.740, and 0.748. A sympathetic reading: if this holds, individual-level prediction from public text is achievable for bounded populations, and the remaining gap to aggregate inference is a calibration problem rather than a modelling impossibility.","feed_headline":"Mixture-of-minds LLM simulator tops individual survey accuracy at 0.775","feed_subtitle":"Anacreon trains one adapter per clustered audience group on public text and beats reported digital-twin scores on ordinal alignment.","key_machinery":"The load-bearing mechanism is a 'mixture of minds': one specialised adapter per cluster of similar individuals, rather than a single dense model for everyone. The pipeline is: (1) a contrastive authorship-representation objective embeds individuals so samples from the same person are pulled together and samples from different people are pushed apart; (2) every record is assigned to its nearest seed individual, partitioning the corpus into clusters; (3) harvested demographics, psychological traits, and mined survey responses are appended to each record, together with a chain-of-emotion, a short trace of emotional states and appraisals from stimulus to response; (4) a separate high-rank LoRA adapter is fine-tuned per cluster on a Gemma 4 12B base with QLoRA, while response-option order is shuffled and non-neutral questions are included in both positive and negative framing; (5) the best checkpoint is selected by maximising $\\alpha A_{\\mathrm{mc}}(\\theta) + \\beta A_{\\mathrm{ord}}(\\theta) - \\gamma \\mathrm{PPL}(\\theta)$ on held-out data. The chain-of-emotion is what conditions the final response on an inferred internal state rather than on the prompt's wording alone.","core_discovery":"The central claim is that individual-level survey responses can be predicted more accurately by a mixture of specialist models than by a single prompted or fine-tuned LLM, and Anacreon is the demonstration on a bounded population: SME merchants who take physical, in-person card payments. In the paper's own terms, the discovery is an extension: clustering a real qualitative corpus around seed people, training one high-rank LoRA adapter per cluster on Gemma 4 12B, augmenting each record with a chain-of-emotion, and selecting checkpoints by a combined score of multiple-choice accuracy, ordinal alignment, and chain-of-emotion perplexity yields a median ordinal alignment of 0.775 and a median top-1 accuracy of 0.679 on a large, externally sourced survey. The paper states that 0.775 is the state of the art on the ordinal-alignment measure, exceeding the reported 0.717, 0.740, and 0.748 of prior digital-twin systems, and that the residual positive bias is small though not eliminated. It frames the result as narrowing, not dissolving, the individual-aggregate gap, with population-level fidelity still to be shown.","pith_inferences":["Beyond the paper: the 0.775 is the median over the 400 retained clusters after a long tail of non-converging clusters was pruned, so an uncurated population would likely score lower unless cluster quality is consistent.","Beyond the paper: because the test questions are mined from the same corpus that supplies training data, some of the gain may reflect familiarity with question format; a genuinely new instrument on the same merchant population is the clean test.","Beyond the paper: the small residual positive bias (+0.437 scale positions) suggests that a simple per-cluster bias-correction layer added at inference could raise ordinal alignment further without changing the architecture."],"forward_implications":["If the reported 0.775 survives clean held-out evaluation, per-cluster fine-tuning on mined public text becomes the strongest reported approach for individual-level survey prediction in a bounded domain.","The recipe—seed clustering, contrastive authorship embedding, attribute mining, per-cluster adapters, chain-of-emotion—can be transferred to other narrow populations that leave enough public text, without needing a single dense retrain.","Because response-option shuffling and mirrored positive/negative framings are part of training, models built this way should be less sensitive to reworded questions than prompted simulators.","The paper's own next step is to post-stratify the per-cluster models to a target population; that is what would convert individual-level fidelity into aggregate prediction."],"supporting_citations":[{"why":"Supplies the Twin-2K-500 dataset, the 0.717 ordinal-alignment baseline, and the human test-retest ceiling used for comparison.","marker":"[37]"},{"why":"Supplies the 0.740 supervised-fine-tuning baseline and the normalised-deviation accuracy convention adopted in eq. (1).","marker":"[18]"},{"why":"Supplies the 0.748 digital-twin mega-study baseline and the finding that twin responses regress toward the population mean on novel stimuli.","marker":"[28]"},{"why":"Supplies the interview-grounded agents whose 0.657 exact-match accuracy Anacreon's 0.679 is compared against, plus a human test-retest reference.","marker":"[26]"},{"why":"Supplies the contrastive authorship-representation objective used to separate individuals before clustering.","marker":"[31]"},{"why":"Supplies the appraisal-based chain-of-emotion construction that Anacreon adapts for its stimulus-to-response traces.","marker":"[7]"},{"why":"Supplies QLoRA, the efficient fine-tuning method used to train each per-cluster adapter.","marker":"[8]"},{"why":"Supplies the Gemma 4 12B base model on which the per-cluster adapters are trained.","marker":"[13]"},{"why":"Documents LLM sycophancy and positivity bias, motivating the mirrored positive/negative framing of training questions.","marker":"[10]"}],"fun_headline_variants":["Anacreon beats digital twins on individual survey alignment","Mixture-of-minds sim hits 0.775 ordinal alignment","Specialist adapters edge out solo LLMs on survey prediction","Anacreon narrows individual-aggregate gap in survey answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the survey questions mined from the same public corpus, and the newest 20% of each cluster's records, are genuinely held out, so the 0.775 score measures prediction of new questions rather than recall of training data—even though those same held-out records are also used to select checkpoints.","fun_headline_variants_meta":{"raw":{"variants":["Anacreon beats digital twins on individual survey alignment","Mixture-of-minds sim hits 0.775 ordinal alignment","Specialist adapters edge out solo LLMs on survey prediction","Anacreon narrows individual-aggregate gap in survey answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2902,"prompt_tokens":1007,"completion_tokens":1895,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":1824}},"tokens_in":623,"tokens_out":1895,"duration_ms":13057,"temperature":1.0,"reasoning_tokens":1824,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:35:53.405075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask Anacreon to answer a newly written survey on the same SME-merchant domain whose questions were not mined from the training corpus, and compare its ordinal alignment on those fresh items with the reported 0.775; a large drop would show the score reflects question-format familiarity rather than generalizable individual prediction. A quicker check is to recompute the score after excluding the newest-20% held-out records from the checkpoint-selection objective in eq. (5).","supporting_citations":[{"cited_title":"QLoRA: Efficient fine- tuning of quantized LLMs","cited_arxiv_id":null,"evidence_quote":"Supplies QLoRA, the efficient fine-tuning method used to train each per-cluster adapter."}],"review_version":1}