{"id":"4228022b-a4a0-4718-ae48-ea907eb625e9","arxiv_id":"2607.10628","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Anamnesis packages backstory-conditioned LLM personas into an interactive open-source survey platform that better matches real human opinion distributions than demographic-list prompting on ATP and New Yorker tasks.","lead":"Anamnesis is an open-source web platform that runs demographically targeted survey simulations by conditioning LLMs on rich narrative backstories rather than short attribute lists. It lets non-technical researchers prototype and stress-test instruments on virtual populations and reports closer matches to real Pew and New Yorker human data than standard persona baselines.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"Table 1 gains rest on an unreproduced matching procedure whose top-M pruning and probabilistic labels may not recover the original Anthology respondent-level pairing.","rationale":"The Reader correctly flags the fidelity of the LLM-generated backstory pool (Appendix A) as the central premise inherited from Anthology/Alterity. That concern is real. The more immediate load-bearing gap for this systems paper, however, is whether the new probabilistic matching + top-M pruning (Algorithm 1, §3.3) preserves the demographic representativeness that made the original gains possible. Table 1 is presented as a successful re-implementation, yet the manuscript supplies no ablation that isolates the matching change from the backstory conditioning itself. The New Yorker result is less sensitive to this issue (no demographic balancing is claimed) but is secondary to the ATP claim that underwrites the abstract. Because the paper already acknowledges related risks and positions itself as an accessibility/systems contribution rather than a new scientific method, the appropriate verdict remains CONDITIONAL; the concrete matching ablation would either clear the residual doubt or force a clearer caveat on the quantitative claims. I therefore agree with the Reader's overall posture and verdict while locating the sharpest untested hinge one step closer to the platform's own algorithmic contribution.","tokens_in":10323,"tokens_out":691,"duration_ms":9658,"concrete_test":"Re-run the three ATP waves of Table 1 on the identical backstory pool twice: once with Algorithm 1 (top-M=50, probabilistic scores) and once with the original Anthology deterministic/Hungarian matching on hard demographic labels (or on argmax of pb,d). Report ΔWD and ΔFro. per wave. If either metric worsens by more than ~10% relative under Algorithm 1, or if cell-level coverage of rare intersections drops materially, the platform's matching step is not fidelity-preserving and the strongest claim must be caveated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim is that Anamnesis produces closer-to-human opinion distributions than BIO/QA baselines (Table 1 WD/Fro. on three ATP waves; New Yorker majority-vote and vote-share). That claim is load-bearing on the assertion that the platform's probabilistic demographic matching (Algorithm 1 with top-M=50 pruning over inferred pb,d(c) labels) recovers a demographically faithful virtual sample comparable to the original Anthology experiments, which used observed human respondent labels and deterministic/Hungarian assignment. Section 3.3 explicitly notes the shift from observed labels to inferred distributions and the pruning step required for interactive latency. If the pruned assignment systematically under-samples low-probability or intersectional cells, or if the inferred labels themselves are skewed, the reported WD/Fro. improvements can be artifacts of a different (and possibly easier) matching regime rather than evidence that the packaged pipeline preserves fidelity. Appendix A already flags training-data skew and shallow binding; the matching change is an additional, unquantified source of the same risk. Without a side-by-side of the same backstory pool under original vs. Algorithm-1 matching, the transfer of the scientific claim to the systems artifact is not fully secured.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"Anamnesis is an open-source web platform that packages the Anthology and Alterity backstory-conditioning methods into a non-technical survey-simulation interface. It provides a graphical survey builder, a pool of ~35K pre-sampled narrative backstories with probabilistic demographic labels, two selection algorithms (Top-K ranking and Balanced Demographic Matching via pruned Hungarian assignment, Algorithm 1), sequential context accumulation, multimodal (image/audio) questions, and post-hoc demographic filtering. Evaluation consists of two case studies: (i) replication of three Pew ATP waves (34, 92, 99) where backstory-conditioned runs report lower Wasserstein distance and Frobenius correlation-matrix distance than BIO/QA demographic-list baselines (Table 1); (ii) a New Yorker Caption Contest multimodal preference task where Anthology improves majority-vote accuracy and human-winner vote share over a zero-shot demographic baseline (with Wilson intervals, McNemar, and paired bootstrap).","tokens_in":10573,"tokens_out":1104,"duration_ms":12421,"significance":"If the reported fidelity gains transfer, the work supplies a transparent, reproducible alternative to closed commercial persona-simulation services and lowers the barrier for social scientists who cannot write sampling scripts. Strengths include an end-to-end open-source system (dispatcher–queue–worker architecture, dual logprobs/N-sample demographic modes, multimodal support), concrete algorithms for probabilistic matching under interactive latency constraints, and external human benchmarks rather than self-defined success metrics. The contribution is primarily systems and packaging rather than a new conditioning theory; its value hinges on whether the platform preserves the scientific claims of the underlying Anthology/Alterity methods under the new probabilistic, pruned matching regime.","major_comments":[{"comment":"§3.3 and Algorithm 1 change the matching regime from the original Anthology setup (observed human respondent labels + deterministic/Hungarian assignment) to inferred probability distributions pb,d(c) with top-M=50 pruning per cell before Hungarian assignment. Table 1’s WD/Fro. gains are the central empirical claim that the platform preserves fidelity, yet the manuscript does not quantify how much of those gains survive under the new matching procedure versus the original respondent-level pairing on the same backstory pool. Without that side-by-side (or an ablation of M and of hard vs. soft labels), it remains possible that the reported improvements partly reflect a different, easier sampling distribution rather than faithful packaging of the prior method.","section":null},{"comment":"Appendix A correctly flags training-data skew and shallow binding as inherited risks, but the evaluation does not stress-test them. Table 1 and the New Yorker study report aggregate distributional match; they do not measure within-group variation, intersectional cell coverage after pruning, or in-group vs. out-group misperception (the Alterity claim the platform claims to operationalize). For a systems paper whose strongest claim is closer-to-human opinion distributions, at least one targeted diagnostic on underrepresented or high-stakes subgroups would make the transfer claim load-bearing rather than assumed.","section":null}],"minor_comments":[{"comment":"Table 1 caption refers readers to Moon et al. (2024) for metric definitions and persona-matching details; a short self-contained restatement of WD and Fro. (and of how ‘max weight’ vs. ‘greedy’ differ from Algorithm 1) would make the table interpretable without the prior paper.","section":null},{"comment":"§4.2 reports Gemini 2.5 Flash at temperature 1.0 with 20 choices per contest; sensitivity to temperature and to the number of samples is not reported, which matters for reproducibility of the majority-vote and vote-share results.","section":null},{"comment":"Figure 2 and §3.2 describe the dispatcher–queue–worker architecture well, but concurrency bounds, failure/retry semantics, and how run-level snapshots guarantee reproducibility under partial worker failure are only sketched.","section":null},{"comment":"Typographical inconsistencies appear throughout (e.g., ‘W e’, ‘Y orker’, ‘A TP’, ‘P arrot’); these look like PDF encoding artifacts and should be cleaned for camera-ready.","section":null},{"comment":"§3.4’s N-sample mode is a practical contribution for users without vLLM logprobs; stating the default N used in any demographic-extension experiments (or noting that none were run for the case studies) would clarify cost/variance trade-offs.","section":null}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems packaging of prior methods and is a reasonable fit for a systems/demo track or a journal that values open research infrastructure. The main risk for a full research venue is that the scientific claim (closer-to-human distributions) is inherited from Anthology/Alterity while the matching substrate has changed without a transfer experiment; if the authors supply that ablation, the contribution becomes much cleaner. Novelty relative to Expected Parrot / Synthetic Users is fairly argued in §5, but the citation set is still heavily self-referential on the conditioning side."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a competent systems paper, not a methods breakthrough. It takes Anthology and Alterity (same group) and ships them as a non-technical web platform with probabilistic demographic matching, sequential context, and multimodal surveys. That packaging is real work and useful.\n\nWhat is actually new is the product surface: survey builder, dispatcher–queue–worker execution, top-K vs balanced matching under inferred pb,d(c) labels (Algorithm 1 with top-M pruning), N-sample demographic mode when logprobs are unavailable, post-hoc filtering, and a New Yorker Caption Contest case study that prior papers did not run. Table 1 re-shows lower WD and Frobenius distance vs BIO/QA on three ATP waves; the New Yorker piece adds majority-vote accuracy, McNemar, and a paired vote-share gain (p=0.0024). Evaluation targets are external human data, so circularity is low. Open-source + GUI is a genuine alternative to closed commercial simulators.\n\nSoft spots, in proportion: the scientific core is inherited. ATP numbers largely re-demonstrate Moon et al. through a new interface. Section 3.3 correctly notes the shift from observed respondent labels to inferred distributions plus top-M=50 pruning for latency; without a side-by-side of original vs Algorithm-1 matching on the same pool, you cannot fully secure that Table 1 gains transfer the original fidelity claim rather than a different matching regime. Appendix A already flags training-data skew and shallow binding—that remains the load-bearing premise, and they own it. Free parameters (M, N, temperature, slot weights) are ordinary systems knobs, not hidden fitting.\n\nWho it is for: computational social scientists and survey methodologists who want to prototype instruments before human fieldwork, and anyone comparing narrative vs short-persona conditioning. Math is light (Hungarian assignment, WD/Fro.); citations are appropriate; code/platform links help reproducibility though the manuscript lacks a commit hash and full dump.\n\nI would send this to peer review as a systems/demo contribution. Expect referees to push on matching fidelity and independent stress-tests of the backstory pool, not to reject for lack of a new conditioning theory. Worth engaging if you care about accessible persona simulation; treat high-stakes demographic claims with the caution the authors themselves write in the appendix.","headline":"Solid systems packaging of Anthology/Alterity into a usable open-source GUI; science is mostly inherited, engineering and multimodal eval are the real additions.","tokens_in":11218,"tokens_out":581,"would_cite":true,"duration_ms":9614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"An open platform turns long narrative backstories into demographically controllable survey simulations that track real opinion data better than short persona prompts.","keywords":["survey simulation","LLM personas","narrative backstories","demographic matching","multimodal surveys","public opinion","open-source platform"],"falsifier":"Re-run the same ATP waves or caption contests after deliberately replacing the narrative pool with backstories known to be demographically skewed or shallowly bound; if Wasserstein and correlation distances no longer improve over short-persona baselines, the central claim fails.","tokens_in":11139,"feed_emoji":"📊","tokens_out":799,"duration_ms":8347,"temperature":0.7,"pith_summary":"Traditional surveys are expensive and hard to target; short demographic prompts to language models produce flattened, stereotypical answers. This paper presents Anamnesis, an open-source web platform that lets non-technical users build multi-question, multimodal surveys and run them on large pools of virtual people. Each virtual person is conditioned on a multi-paragraph life narrative rather than a bullet list of attributes, answers questions sequentially so prior answers stay in context, and can be selected so the sample matches a target demographic distribution under probabilistic labels. Across replications of Pew political and biomedical waves and a New Yorker caption-preference task, the resulting answer distributions sit closer to real human data than standard persona baselines. The practical claim is that researchers can prototype and stress-test instruments on balanced virtual populations before recruiting a single human subject, with full transparency and reproducibility.","feed_headline":"Narrative backstories beat short personas on real survey data","feed_subtitle":"Open platform lets non-technical users run demographically balanced virtual polls that track human answers more closely","key_machinery":"Backstory-conditioned simulation: each virtual respondent is a multi-paragraph life narrative (pre-sampled and demographically labeled) that is prepended to the survey; answers accumulate sequentially, and a top-M-pruned Hungarian matching step selects a demographically balanced sample under probabilistic labels.","core_discovery":"When language-model respondents are conditioned on rich narrative backstories, selected by probabilistic demographic matching, and allowed to accumulate answer context across questions, the aggregate opinion distributions on both text-only public-opinion items and multimodal caption preference more closely match real human survey data than short demographic-list prompting.","pith_inferences":["If the matching and sequential-context machinery transfer, the same platform could serve as a cheap pre-registration check for power and item wording before expensive field studies.","The probabilistic demographic layer suggests a natural extension: continuous rather than categorical attributes (income, ideology scores) without redesigning the selection algorithm.","Persistent gaps versus human baselines on sensitive political items would quantify remaining stereotype risk more precisely than average Wasserstein distance alone."],"forward_implications":["A social scientist can draft a multi-question instrument, set target demographics, and obtain approximate response distributions before recruiting humans.","Multimodal items (images, audio) can be stress-tested on the same demographically balanced virtual populations used for text surveys.","Post-hoc demographic filtering of a single large run replaces the need to re-sample and re-run for every subgroup comparison.","Open, reproducible code and a GUI lower the barrier relative to closed commercial simulation services and to pure code libraries."],"fun_headline_variants":["Rich backstories beat short personas on real survey matches","Narrative conditioning tracks human opinions better than personas","Backstory-conditioned LLMs closer to real polls than persona lists","Demographically matched backstories outperform short personas","Open Anamnesis yields survey distributions nearer human data"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The premise that the pre-sampled pool of model-generated life narratives, once demographically matched and given sequential context, produces authentic in-group opinions rather than training-data stereotypes or out-group caricatures.","fun_headline_variants_meta":{"raw":{"variants":["Rich backstories beat short personas on real survey matches","Narrative conditioning tracks human opinions better than personas","Backstory-conditioned LLMs closer to real polls than persona lists","Demographically matched backstories outperform short personas","Open Anamnesis yields survey distributions nearer human data"]},"model":"grok-4.5","effort":"low","cost_usd":0.003366,"raw_usage":{"total_tokens":1069,"prompt_tokens":714,"num_sources_used":0,"completion_tokens":63,"cost_in_usd_ticks":33660000,"prompt_tokens_details":{"text_tokens":714,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":292,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":714,"tokens_out":63,"duration_ms":4030,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T10:21:24.946396+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the same ATP waves or caption contests after deliberately replacing the narrative pool with backstories known to be demographically skewed or shallowly bound; if Wasserstein and correlation distances no longer improve over short-persona baselines, the central claim fails.","supporting_citations":[],"review_version":1}