Pith. sign in

REVIEW 2 major objections 5 minor 19 references

An open platform turns long narrative backstories into demographically controllable survey simulations that track real opinion data better than short persona prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 10:21 UTC pith:4G5QT7F3

load-bearing objection Solid systems packaging of Anthology/Alterity into a usable open-source GUI; science is mostly inherited, engineering and multimodal eval are the real additions. the 2 major comments →

arxiv 2607.10628 v1 pith:4G5QT7F3 submitted 2026-07-12 cs.CL cs.AIcs.HC

Anamnesis: An Open-Source Platform for Large-Scale Backstory-Conditioned Survey Simulation

Song-Ze Yu , Joseph Suh , Serina Chang , David M. Chan This is my paper
classification cs.CL cs.AIcs.HC
keywords survey simulationLLM personasnarrative backstoriesdemographic matchingmultimodal surveyspublic opinionopen-source platform
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Traditional surveys are expensive and hard to target; short demographic prompts to language models produce flattened, stereotypical answers. This paper presents Anamnesis, an open-source web platform that lets non-technical users build multi-question, multimodal surveys and run them on large pools of virtual people. Each virtual person is conditioned on a multi-paragraph life narrative rather than a bullet list of attributes, answers questions sequentially so prior answers stay in context, and can be selected so the sample matches a target demographic distribution under probabilistic labels. Across replications of Pew political and biomedical waves and a New Yorker caption-preference task, the resulting answer distributions sit closer to real human data than standard persona baselines. The practical claim is that researchers can prototype and stress-test instruments on balanced virtual populations before recruiting a single human subject, with full transparency and reproducibility.

Core claim

When language-model respondents are conditioned on rich narrative backstories, selected by probabilistic demographic matching, and allowed to accumulate answer context across questions, the aggregate opinion distributions on both text-only public-opinion items and multimodal caption preference more closely match real human survey data than short demographic-list prompting.

What carries the argument

Backstory-conditioned simulation: each virtual respondent is a multi-paragraph life narrative (pre-sampled and demographically labeled) that is prepended to the survey; answers accumulate sequentially, and a top-M-pruned Hungarian matching step selects a demographically balanced sample under probabilistic labels.

Load-bearing premise

The premise that the pre-sampled pool of model-generated life narratives, once demographically matched and given sequential context, produces authentic in-group opinions rather than training-data stereotypes or out-group caricatures.

What would settle it

Re-run the same ATP waves or caption contests after deliberately replacing the narrative pool with backstories known to be demographically skewed or shallowly bound; if Wasserstein and correlation distances no longer improve over short-persona baselines, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A social scientist can draft a multi-question instrument, set target demographics, and obtain approximate response distributions before recruiting humans.
  • Multimodal items (images, audio) can be stress-tested on the same demographically balanced virtual populations used for text surveys.
  • Post-hoc demographic filtering of a single large run replaces the need to re-sample and re-run for every subgroup comparison.
  • Open, reproducible code and a GUI lower the barrier relative to closed commercial simulation services and to pure code libraries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the matching and sequential-context machinery transfer, the same platform could serve as a cheap pre-registration check for power and item wording before expensive field studies.
  • The probabilistic demographic layer suggests a natural extension: continuous rather than categorical attributes (income, ideology scores) without redesigning the selection algorithm.
  • Persistent gaps versus human baselines on sensitive political items would quantify remaining stereotype risk more precisely than average Wasserstein distance alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. Anamnesis is an open-source web platform that packages the Anthology and Alterity backstory-conditioning methods into a non-technical survey-simulation interface. It provides a graphical survey builder, a pool of ~35K pre-sampled narrative backstories with probabilistic demographic labels, two selection algorithms (Top-K ranking and Balanced Demographic Matching via pruned Hungarian assignment, Algorithm 1), sequential context accumulation, multimodal (image/audio) questions, and post-hoc demographic filtering. Evaluation consists of two case studies: (i) replication of three Pew ATP waves (34, 92, 99) where backstory-conditioned runs report lower Wasserstein distance and Frobenius correlation-matrix distance than BIO/QA demographic-list baselines (Table 1); (ii) a New Yorker Caption Contest multimodal preference task where Anthology improves majority-vote accuracy and human-winner vote share over a zero-shot demographic baseline (with Wilson intervals, McNemar, and paired bootstrap).

Significance. If the reported fidelity gains transfer, the work supplies a transparent, reproducible alternative to closed commercial persona-simulation services and lowers the barrier for social scientists who cannot write sampling scripts. Strengths include an end-to-end open-source system (dispatcher–queue–worker architecture, dual logprobs/N-sample demographic modes, multimodal support), concrete algorithms for probabilistic matching under interactive latency constraints, and external human benchmarks rather than self-defined success metrics. The contribution is primarily systems and packaging rather than a new conditioning theory; its value hinges on whether the platform preserves the scientific claims of the underlying Anthology/Alterity methods under the new probabilistic, pruned matching regime.

major comments (2)
  1. §3.3 and Algorithm 1 change the matching regime from the original Anthology setup (observed human respondent labels + deterministic/Hungarian assignment) to inferred probability distributions pb,d(c) with top-M=50 pruning per cell before Hungarian assignment. Table 1’s WD/Fro. gains are the central empirical claim that the platform preserves fidelity, yet the manuscript does not quantify how much of those gains survive under the new matching procedure versus the original respondent-level pairing on the same backstory pool. Without that side-by-side (or an ablation of M and of hard vs. soft labels), it remains possible that the reported improvements partly reflect a different, easier sampling distribution rather than faithful packaging of the prior method.
  2. Appendix A correctly flags training-data skew and shallow binding as inherited risks, but the evaluation does not stress-test them. Table 1 and the New Yorker study report aggregate distributional match; they do not measure within-group variation, intersectional cell coverage after pruning, or in-group vs. out-group misperception (the Alterity claim the platform claims to operationalize). For a systems paper whose strongest claim is closer-to-human opinion distributions, at least one targeted diagnostic on underrepresented or high-stakes subgroups would make the transfer claim load-bearing rather than assumed.
minor comments (5)
  1. Table 1 caption refers readers to Moon et al. (2024) for metric definitions and persona-matching details; a short self-contained restatement of WD and Fro. (and of how ‘max weight’ vs. ‘greedy’ differ from Algorithm 1) would make the table interpretable without the prior paper.
  2. §4.2 reports Gemini 2.5 Flash at temperature 1.0 with 20 choices per contest; sensitivity to temperature and to the number of samples is not reported, which matters for reproducibility of the majority-vote and vote-share results.
  3. Figure 2 and §3.2 describe the dispatcher–queue–worker architecture well, but concurrency bounds, failure/retry semantics, and how run-level snapshots guarantee reproducibility under partial worker failure are only sketched.
  4. Typographical inconsistencies appear throughout (e.g., ‘W e’, ‘Y orker’, ‘A TP’, ‘P arrot’); these look like PDF encoding artifacts and should be cleaned for camera-ready.
  5. §3.4’s N-sample mode is a practical contribution for users without vLLM logprobs; stating the default N used in any demographic-extension experiments (or noting that none were run for the case studies) would clarify cost/variance trade-offs.

Circularity Check

0 steps flagged

No load-bearing circularity: platform packages prior methods and is scored against external human survey/preference data, not against quantities defined by its own fits.

full rationale

Anamnesis is a systems paper that operationalizes the Anthology/Alterity backstory-conditioning pipeline into a GUI with probabilistic matching (Algorithm 1) and sequential context accumulation. Its strongest claims are empirical: lower Wasserstein and Frobenius distances to real Pew ATP response distributions than BIO/QA persona baselines (Table 1), and higher majority-vote accuracy plus human-winner vote share on the New Yorker Caption Contest. Both targets are external human data collected independently of the platform; success is not defined by re-deriving a quantity already fitted inside the same run. Self-citations to Moon et al. (2024) and Kang et al. (2025) supply the methods being packaged and are normal for an operationalization paper; those methods themselves were scored on the same external benchmarks and are re-checked here. There is no self-definitional loop, no fitted parameter renamed as a prediction, no uniqueness theorem imported to forbid alternatives, and no ansatz smuggled via citation that forces the reported gains by construction. Minor residual risk (training-data skew of the backstory pool, top-M pruning of the assignment) is acknowledged in Appendix A as a limitation of the underlying methodology, not as a circular reduction. Score 1 only for ordinary self-citation of the packaged frameworks; the central evaluation chain remains non-circular.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The platform’s claims rest on inherited methodological assumptions from Anthology/Alterity plus a small set of engineering hyperparameters chosen for interactive latency. No new physical or mathematical entities are postulated; the free parameters are implementation knobs (pruning width, sampling temperature, N-sample count) that affect which personas are selected and how answers are drawn but are not fitted to the ATP or New Yorker targets themselves.

free parameters (4)
  • top-M candidates per demographic cell = 50
    Default M=50 used to prune the candidate pool before Hungarian assignment so that client-side matching stays interactive; choice is latency-driven, not derived from data.
  • Gemini temperature for New Yorker trials = 1.0
    Temperature set to 1.0 for the 20 choices per contest; affects diversity of sampled preferences.
  • N in N-sample demographic mode
    Number of repeated draws used to estimate categorical distributions when log-probabilities are unavailable; larger N reduces variance at higher cost.
  • sample size S and slot allocation weights
    User-chosen number of virtual respondents and optional non-uniform cell weights; free design parameters of each simulation run.
axioms (4)
  • domain assumption Narrative backstories generated from American Voices Project-style interview prompts, when used as conditioning context, produce more representative and consistent survey responses than short demographic attribute lists.
    Inherited from Moon et al. (2024) Anthology; treated as given and re-tested only via the platform’s ATP replications.
  • domain assumption Inferred probabilistic demographic labels obtained by querying an LLM about a backstory are accurate enough for top-K or Hungarian matching to approximate a target population distribution.
    Stated in §3.3–3.4; no independent human validation of the inferred pb,d(c) distributions is provided in this paper.
  • domain assumption Sequential accumulation of question–answer pairs inside a single persona’s context window improves cross-question belief consistency.
    Default answering algorithm (§3.5); supported by prior Anthology results and by the Frobenius-norm metric here.
  • ad hoc to paper Hungarian assignment on a pruned candidate matrix of size S×|C| with S≤50 yields demographically balanced samples of acceptable quality for interactive use.
    Engineering approximation introduced in Algorithm 1 to keep latency low; optimality is sacrificed for speed.

pith-pipeline@v1.1.0-grok45 · 14339 in / 3327 out tokens · 72474 ms · 2026-07-14T10:21:24.946396+00:00 · methodology

0 comments
read the original abstract

We present Anamnesis, an interactive system for demographically controllable survey simulation using large language models. Open-source, and designed for non-technical users/researchers, Anamnesis enables the prototyping and stress-testing of survey instruments on virtual populations rather than real human subjects. The platform operationalizes the recently introduced Anthology and Alterity frameworks, which use structured narrative backstories to condition model responses, within a unified web interface. It supports open-ended generation, probabilistic demographic resampling, and multimodal (image and audio) surveys. We evaluate the system through two case studies: (1) replicating segments of Pew Research Center's American Trends Panel (ATP) on political typology and biomedical issues and (2) emulating human preference in the New Yorker Caption Contest. In both cases, Anamnesis produces opinion distributions that more closely match real-world survey data than standard persona-prompting baselines, offering a transparent, reproducible, and open-source alternative to proprietary simulation services.

Figures

Figures reproduced from arXiv: 2607.10628 by David M. Chan, Joseph Suh, Serina Chang, Song-Ze Yu.

Figure 1
Figure 1. Figure 1: Anamnesis is an interactive system for demo￾graphically controllable survey simulation using large lan￾guage models. It provides a non-technical interface for An￾thology, a method which approximates large-scale human studies by conditioning LLMs to representative, consistent, and diverse virtual personas. Together, these systems enable rapid prototyping and stress-testing of survey instruments on diverse v… view at source ↗
Figure 2
Figure 2. Figure 2: System overview of Anamnesis. Pre-sampled backstories generated via Anthology are stored as a persona pool, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 3 linked inside Pith

  1. [1]

    Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    Virtual personas for language models via an anthology of backstories , author=. Proceedings of the 2024 conference on empirical methods in natural language processing , pages=

    2024

  2. [2]

    Second Conference on Language Modeling , year=

    Deep Binding of Language Model Virtual Personas: a Study on Approximating Political Partisan Misperceptions , author=. Second Conference on Language Modeling , year=

  3. [3]

    International conference on machine learning , pages=

    Whose opinions do language models reflect? , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  4. [4]

    American Voices Project Methodology , year =

  5. [5]

    Nature Machine Intelligence , volume=

    Large language models that replace human participants can harmfully misportray and flatten identity groups , author=. Nature Machine Intelligence , volume=. 2025 , publisher=

    2025

  6. [6]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    CoMPosT: Characterizing and evaluating caricature in LLM simulations , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  7. [7]

    2026 , organization =

    Expected Parrot , author =. 2026 , organization =

    2026

  8. [8]

    2026 , organization =

    Artificial Societies — Company Profile , author =. 2026 , organization =

    2026

  9. [9]

    2026 , organization =

    Synthetic Users — Company Profile , author =. 2026 , organization =

    2026

  10. [10]

    America Trends Panel Waves , author =

  11. [11]

    Few-shot personalization of llms with mis-aligned responses , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  12. [12]

    CLAIR: Evaluating Image Captions with Large Language Models

    Chan, David M and Petryk, Suzanne and Gonzalez, Joseph E and Darrell, Trevor and Canny, John. CLAIR: Evaluating Image Captions with Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023

    2023

  13. [13]

    arXiv preprint arXiv:2503.16527 , year=

    Llm generated persona is a promise with a catch , author=. arXiv preprint arXiv:2503.16527 , year=

    Pith/arXiv arXiv

  14. [14]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Aligning language models to user opinions , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  15. [15]

    arXiv preprint arXiv:2411.10109 , year=

    Generative agent simulations of 1,000 people , author=. arXiv preprint arXiv:2411.10109 , year=

    Pith/arXiv arXiv

  16. [16]

    Understanding

    Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest , author=. arXiv preprint arXiv:2209.06293 , year=

    Pith/arXiv arXiv

  17. [17]

    2020 , url=

    Jain, Lalit and Jamieson, Kevin and Mankoff, Robert and Nowak, Robert and Sievert, Scott , title=. 2020 , url=

    2020

  18. [18]

    Humor in Collective Discourse: Unsupervised Funniness Detection in The New Yorker Cartoon Caption Contest

    Radev, Dragomir and Stent, Amanda and Tetreault, Joel and Pappu, Aasish and Iliakopoulou, Aikaterini and Chanfreau, Agustin and de Juan, Paloma and Vallmitjana, Jordi and Jaimes, Alejandro and Jha, Rahul and Mankoff, Robert. Humor in Collective Discourse: Unsupervised Funniness Detection in The New Yorker Cartoon Caption Contest. LREC. 2016

    2016

  19. [19]

    KDD , year=

    Inside jokes: Identifying humorous cartoon captions , author=. KDD , year=