Co-design of LLM-based preference agents: participation may drive overtrust

Michael J. Fell

REVIEW 3 major objections 4 minor 8 references

Co-design of LLM-based preference agents: participation may drive overtrust

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Co-designing an LLM-based preference agent can build trust while hiding systematic errors, because the process itself—not just the model—creates the feeling of being represented.

desk verdict A careful qualitative study with a plausible but not yet secured causal claim: co-design may indeed breed overtrust, but the design cannot isolate participation from generic LLM personalization. read the letter →

arxiv 2607.21757 v1 pith:YZQHZ4F5 submitted 2026-07-23 cs.CY cs.AIcs.HC

Michael J. Fell This is my paper

classification cs.CYcs.AIcs.HC

keywords LLMagentsco-designparticipatoryAIpreferencesimulationalgorithmicfidelityovertrusthomeenergyalignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether co-designing a personal LLM-based preference agent with the person it represents actually makes the agent more accurate. In a study of 12 UK participants building agents for household-energy decisions, people engaged enthusiastically and mostly concluded their agents represented them well. But independent validation on new scenarios showed mixed alignment: agent responses were more homogeneous, more decisive, more abstract, and never chose the "neither" option that humans sometimes did. The author argues that participation and process transparency can act as an "overtrust engine"—a combination of limited testing, the Barnum effect, positivity bias, and social-desirability bias—that produces trust while hiding misalignment visible only in aggregate. If this is right, co-design's main effect may be to make users feel well represented rather than to make agents accurate, which matters wherever such agents are used to stand in for human preferences in research, policy, or markets.

What carries the argument

The central object is the co-designed personal agent description, a second-person persona built from survey data and interview refinements. The named mechanism is the "overtrust engine": the author's term for the combined dynamics—testing only on scenarios that are refined until they look aligned, Barnum-effect acceptance of general statements as personal, positivity and salience bias, and the IKEA-effect-style investment in something one helped create—that turn participation and process transparency into a source of confidence rather than scrutiny. This mechanism carries the paper's argument because it explains the gap between participants' high perceived fidelity and the independent valida

What would settle it

A direct test would compare one group that co-designs an agent with a matched group that receives an equally personalised agent built without their participation; if trust and perceived fidelity are the same in both groups, the overtrust engine is not specifically driven by co-design. A second check: if independent alignment turns out high for co-designed agents on familiar, well-covered topics, the claim of systematic misalignment would be weakened.

Watch

Extended reading notes

Core claim

The paper's central claim is that participation and process transparency in co-designing an LLM-based preference agent function as an "overtrust engine": they make the user feel the agent represents them accurately while hiding systematic misalignment that is only visible at the group level. In the reported study, 12 participants co-designed agents for household-energy decisions through a background survey, an interview, and a validation survey; 10 of 12 strongly agreed the final agent did a good job representing their preferences, yet independent scenario testing showed mixed human-agent alignment, with agents never choosing the neutral options humans sometimes chose, and being more homogen

Load-bearing premise

The load-bearing premise is that co-design itself—not generic LLM persona behaviour or the study's format—causes the elevated trust, and the study did not include a control condition with equally personalised agents that participants did not shape.

Editorial extensions

If this is right

People who co-design a preference agent are likely to overestimate how well it represents them, even when independent validation shows mixed accuracy.
Agent outputs in this setting tend to be more homogeneous, more decisive, and more abstract than the human responses they stand in for, so simulations built this way will flatten the diversity of human preferences.
The misalignment is invisible from any single user's vantage point; it can only be detected by comparing many agents' responses with many human responses.
Deployed at scale, overtrusted agents could skew energy, research, and policy decisions toward model defaults while each user believes their own interests are being served.
Co-design still provides value—it keeps humans involved, lets misrepresentations be challenged, and is a rich data-collection method—but it does not by itself ensure alignment; ongoing validation and human oversight are prerequisites.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the paper is right, the same overtrust engine should appear in other domains where people hold weakly formed preferences, such as personal finance or health decisions; a controlled replication there would test the mechanism's generality.
Because the systematic biases only show up in aggregate, a practical safeguard would be a 'diversity dashboard' that routinely compares agent-response distributions with human-response distributions on probe scenarios, flagging homogenisation before deployment.
The study's qualitative design cannot separate the participation effect from the Barnum effect; a matched non-participatory personalisation arm would settle whether co-design adds overtrust beyond generic personalised LLM output.
Researchers who use co-designed agents as stand-ins for human samples should treat participants' self-reported fidelity as evidence about the relationship built by the process, not as evidence about predictive accuracy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real contribution here is the gap between perceived and independently-assessed alignment: participants who co-designed a preference agent reported high trust and fidelity even when fresh-scenario validation showed mixed, homogeneous, and systematically skewed agent outputs. That demonstration is worth having, and the "overtrust engine" framing is a useful way to package known effects (Barnum, IKEA, positivity bias, limited testing) into a mechanism that future work can test.

What the paper does well: it is honest and well-documented. The method is laid out in detail, with reporting guidelines followed, limitations stated openly, and claims hedged. The observation that agents were more homogeneous, more decisive, and more abstract than the humans is a concrete addition to the LLM-persona critique. The co-design interview itself is also an interesting data-collection method, and the author is careful not to treat the validation survey as a stable ground truth given preference plasticity.

The soft spot is the one the stress-test identifies, and it is load-bearing. The title says participation may drive overtrust, but there is no non-participatory control arm. Every component of the proposed engine — Barnum effect, positivity bias, social desirability, confidence in a personalized-sounding output — could operate with a survey-derived persona the user never shaped. The paper's own limitation section (5.1) concedes that participant control was reasonably limited, which further blurs the "participation" component. The author is transparent about this, but the central inference is nevertheless underdetermined. Minor weaknesses: a single proprietary LLM, a single researcher doing the coding with no inter-rater reliability check, and no data sharing due to ethics. For a small qualitative study these are acceptable, but they cap the confidence one can place in the specifics.

Who this is for: people working on LLM-based preference simulation, participatory AI, and human-agent trust. It is a hypothesis-generating paper, not a demonstrated causal effect, and should be read and cited as such.

Recommendation: send it to peer review. The study is carefully done, the question matters, and a good referee can push for either a softened framing or a follow-up with a non-participatory comparator. Worth engaging, not worth treating as settled.

Referee Report

3 major / 4 minor

Summary. The paper reports a primarily qualitative study (N=12) in which participants co-designed LLM-based personal preference agents for household energy decisions via a background survey, a co-design interview, and a validation survey. Participants generally reported that their agents represented them well and expressed trust, while independent scenario-based validation showed mixed alignment and revealed that agent responses were more homogeneous, decisive, and abstract than human responses. The paper argues that participation and process transparency can function as an 'overtrust engine': a combination of limited testing, the Barnum effect, positivity bias, and social desirability that promotes trust while concealing systematic misalignment. The author proposes that alignment should be viewed not as a fixed state but as enacted through the co-design process, and discusses implications for research and deployment.

Significance. If the proposed mechanism is accepted, the paper makes a useful and timely contribution to the participatory AI and LLM-agent literature. Its strengths are its honesty about limitations, its explicit engagement with preference plasticity, the qualitative richness of the interviews, and the clear articulation of an aggregate-invisibility problem: systematic biases in agent outputs may be invisible to any individual user while producing structural consequences at scale. The paper does not overclaim statistical generalization and appropriately labels its quantitative comparisons as descriptive. The main value is conceptual: it offers a testable hypothesis that co-design and process transparency may increase trust without increasing objective alignment. However, the central causal interpretation is underdetermined by the design, and the paper would need to be reframed or supplemented before the 'overtrust engine' can be regarded as established rather than as a plausible interpretive hypothesis.

major comments (3)

[Abstract; Section 5.2.2] The central claim that co-design/process transparency drives overtrust lacks a non-participatory control condition. Every mechanism invoked in Section 5.2.2 — limited testing, Barnum/Forer effects, positivity bias, social desirability, and the IKEA effect — could plausibly operate for any personalized LLM agent even without co-design. A participant receiving a survey-derived persona and a few well-aligned example responses may likewise overestimate fidelity. The paper's own Section 5.1 acknowledges that participants' genuine steering was 'reasonably limited,' which further weakens the attribution to co-design specifically. The observed gap between perceived and independently-assessed alignment is consistent with the proposed mechanism, but it does not distinguish it from generic LLM personalization effects. This is load-bearing because the abstract and conclusions assert that participati
[Section 5.2.2; Figure 5] The 'overtrust engine' is introduced as a 'powerful combination' of four or five mechanisms, but the study does not contain evidence that these mechanisms combine, that they are produced by co-design rather than by the study procedure, or that they jointly constitute a single engine. The qualitative data illustrate each mechanism in isolation, but the inference that co-design 'produced the conditions through which alignment came to be perceived' goes beyond what the data can support. In particular, participants never saw the independent validation results, so their continued trust may reflect an artifact of the study's staged feedback design rather than a stable property of participatory processes. The paper should either present the overtrust engine as an explicitly exploratory model requiring dedicated testing, or provide evidence that the mechanisms co-occur and interact as claimed.
[Section 5.2.2; Section 4.5] The 'independent validation' is presented as a benchmark against which perceived alignment is compared, but the paper itself acknowledges in Section 5.2.2 that human preferences on unfamiliar topics are plastic and context-dependent. This undercuts the status of the validation survey as a stable ground truth. The systematic character of agent outputs (homogeneity, decisiveness, abstractness) is less vulnerable to this objection, but the quantitative gap between perceived and independent alignment is contingent on the one-shot survey responses. The author partially addresses this by saying the issue is not that perceived alignment was high while 'real' alignment was low, but the subtlety is not consistently maintained in the abstract and conclusions, which speak of 'mixed human-agent alignment' and 'independent validation.' The distinction should be carried through the entire framing, and

minor comments (4)

[Section 4.5, Figure 3 caption] The caption states 'Scenario 3 not included as it has a numerical response,' but the text later describes scenario 3's numeric outcomes in detail. Clarify that the figure excludes it for scaling reasons, while the text discusses it separately.
[Section 3.2] The thematic analysis was conducted by the author alone with no mention of inter-coder reliability or independent audit. For a study whose central claims depend on interpretive coding, a brief statement on coding checks or member checking would strengthen transparency.
[Section 5.4] The mitigation suggestions are reasonable but are presented as if they follow directly from the findings. Several, such as 'framing the agent as a well-acquainted advisor,' are not tested in this study and should be labeled as speculative design directions rather than evidence-based recommendations.
[Section 8] The data availability statement says no data will be shared. This is understandable given the personal nature of the data and ethics approval, but for a qualitative study with small N and interpretive coding, it limits readers' ability to assess the analysis. At minimum, the author could provide the full coding framework (already in S8) with more extensive anonymized quote excerpts than are currently included.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper's central claim is an interpretive synthesis of independent observations, not a derivation from fitted inputs or self-citation.

full rationale

This paper contains no mathematical derivation, parameter fitting, or uniqueness theorem, so the classic circularity patterns (defined-in-terms-of, fitted-input-called-prediction, self-citation chains, ansatz-via-citation) do not apply. The empirical result is a straightforward contrast between two independently collected sets of observations: participants' self-reported trust in their co-designed agents (e.g., 10/12 strongly agreeing the agent represented them) and the researcher's independent validation comparing participant and agent responses to new scenarios (showing mixed alignment and agent homogeneity). The 'overtrust engine' is an explanatory construct assembled from those observations plus external, well-established psychological concepts (Barnum effect, IKEA effect, positivity bias, social desirability), cited to independent literatures rather than to the author's own prior work. The paper is candid that its limitations make causal attribution uncertain: it notes that participant control was 'reasonably limited' and that the co-design process 'had no mechanism for conveying the boundaries of tested alignment.' Concerns that the study lacks a non-participatory control arm, and that generic LLM personalization or the Barnum effect could explain the trust gap, are validity/identifiability concerns about the causal claim, not circularity: the paper does not assume the truth of its conclusion in its evidence or method. The one mention of a 'somewhat circular approach' in the literature review concerns prior modelled-preference datasets (J.-N. Li et al. 2025; Poddar et al. 2024) and is not a step in the present paper's argument. Overall, the derivation chain is self-contained qualitative inference, so the circularity score is appropriately low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central argument rests on assumptions about what the self-report measures, the validity of the validation scenarios, generalization from one LLM, and single-researcher coding. No free parameters are fitted; the paper is qualitative.

assumptions (4)

domain assumption Participants' expressed trust and sentiment ratings reflect genuine psychological trust rather than politeness or demand characteristics.
The study relies on self-report in interviews and surveys (Section 4.4) without behavioral measures or triangulation; social desirability is mentioned as a possible confound.
domain assumption The validation survey scenarios are a meaningful test of alignment for the co-designed agents.
Section 5.1 notes scenarios were unfamiliar and views plastic; the paper acknowledges this but still uses them to establish 'independent' misalignment.
domain assumption GPT-5's homogeneity and decisiveness generalize to other LLMs used for preference agents.
Section 5.1 admits only one model was tested; generalization relies on prior literature.
domain assumption The researcher's single-authored thematic coding and sentiment classification are reliable.
Section 3.2 describes coding by the author alone in NVivo; no inter-rater reliability is reported.

invented entities (1)

'Overtrust engine' mechanism
purpose: Conceptual mechanism explaining how co-design participation and process transparency increase trust in LLM preference agents while hiding systematic misalignment.
The paper presents qualitative evidence consistent with the mechanism but no independent falsifiable operationalization; contributing effects (Barnum, IKEA, social desirability) are borrowed from prior literature.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Co-design of LLM-based preference agents: participation may drive overtrust." pith.science (2026). https://pith.science/paper/YZQHZ4F5

@misc{pith2026260721757,
  author       = {Pith},
  title        = {Pith review of: Co-design of LLM-based preference agents: participation may drive overtrust},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YZQHZ4F5}},
  note         = {Machine review of arXiv:2607.21757}
}

read the original abstract

Large language models are increasingly used to simulate human preferences in research and practical applications, raising concerns about validation, misrepresentation, and exclusion. Co-designing agents with the people they represent is a promising way to address these concerns, but participation may also mask the problems it appears to solve. This paper explores that tension through a primarily qualitative study in which 12 participants co-designed personal preference agents in the domain of household energy, via a background survey, co-design interview, and validation survey. Participants engaged readily and mostly came to see their agents as representing them well. Independent validation, however, revealed mixed human-agent alignment, with agent responses markedly more homogeneous, decisive, and abstract than the human sample. I argue that participation and process transparency can act as an "overtrust engine" that promotes trust while concealing systematic misalignment with potential structural consequences at scale. I develop this as a core mechanism in participatory preference agent design, treating individual alignment not as a fixed state but as an enacted process.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 2 linked inside Pith

[1]

General discussion about your use and views on AI, and about potentially being represented by it
[2]

Refining and testing the agent description created from your survey responses
[3]

budget" tariff. Imagine a supplier is planning to offer a

Your thoughts on the process and on future uses of agents. We will be using a large language model, similar to ChatGPT or Claude, which I will operate during our conversation. I will handle all the technical aspects while you guide the content and decisions. Bear with me if it is a bit slow. A.2 Warnings and considerations • We will store the agent descri...

2026
[4]

https://doi.org/10.48550/arXiv.2411.10109 Poddar, S., Wan, Y., Ivison, H., Gupta, A., Jaques, N., 2024

Generative Agent Simulations of 1,000 People. https://doi.org/10.48550/arXiv.2411.10109 Poddar, S., Wan, Y., Ivison, H., Gupta, A., Jaques, N., 2024. Personalizing reinforcement learning from human feedback with variational preference learning, in: Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24. Curran ...

arXiv doi:10.48550/arxiv.2411.10109 2024
[18]

You are…

https://doi.org/10.1080/15710880701875068 Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., Hashimoto, T., 2023. Whose opinions do language models reflect?, in: Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, Honolulu, Hawaii, USA, pp. 29971–30004. Satre-Meloy, A., Hampton, S., 2024. Physical, socio -psych...

arXiv doi:10.48550/arxiv.2210.01790 2023
[517]

What Is Codesign? [WWW Document]

https://doi.org/10.1080/08874417.2025.2483832 IxDF, 2026. What Is Codesign? [WWW Document]. IxDF - Interaction Design Foundation. URL https://ixdf.org/literature/topics/codesign (accessed 6.30.26). Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Vierling, L., Hong, D., Zhou, J., Zhang, Z., Zeng, F., Dai, J., Pan, X., Ng, K.Y., O...

arXiv 2025
[2024]

Grand Challenge

The illusion of artificial inclusion. https://doi.org/10.1145/3613904.3642703 Argyle, L.P., Busby, E.C., Fulda, N., Gubler, J.R., Rytting, C., Wingate, D., 2023. Out of One, Ma ny: Using Language Models to Simulate Human Samples. Polit. Anal. 31, 337 –351. https://doi.org/10.1017/pan.2023.2 Bommasani, R., Creel, K.A., Kumar, A., Jurafsky, D., Liang, P., 2...

arXiv 2023
[2025]

https://doi.org/10.48550/arXiv.2510.22954 Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A., 2024a

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). https://doi.org/10.48550/arXiv.2510.22954 Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A., 2024a. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nat Mach Intell 6, 383 –392. https://doi.org/10.1038/s42256-024-00820-y Kir...

doi:10.48550/arxiv.2510.22954 2025

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

General discussion about your use and views on AI, and about potentially being represented by it

[2] [2]

Refining and testing the agent description created from your survey responses

[3] [3]

budget" tariff. Imagine a supplier is planning to offer a

Your thoughts on the process and on future uses of agents. We will be using a large language model, similar to ChatGPT or Claude, which I will operate during our conversation. I will handle all the technical aspects while you guide the content and decisions. Bear with me if it is a bit slow. A.2 Warnings and considerations • We will store the agent descri...

2026

[4] [4]

https://doi.org/10.48550/arXiv.2411.10109 Poddar, S., Wan, Y., Ivison, H., Gupta, A., Jaques, N., 2024

Generative Agent Simulations of 1,000 People. https://doi.org/10.48550/arXiv.2411.10109 Poddar, S., Wan, Y., Ivison, H., Gupta, A., Jaques, N., 2024. Personalizing reinforcement learning from human feedback with variational preference learning, in: Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24. Curran ...

arXiv doi:10.48550/arxiv.2411.10109 2024

[5] [18]

You are…

https://doi.org/10.1080/15710880701875068 Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., Hashimoto, T., 2023. Whose opinions do language models reflect?, in: Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, Honolulu, Hawaii, USA, pp. 29971–30004. Satre-Meloy, A., Hampton, S., 2024. Physical, socio -psych...

arXiv doi:10.48550/arxiv.2210.01790 2023

[6] [517]

What Is Codesign? [WWW Document]

https://doi.org/10.1080/08874417.2025.2483832 IxDF, 2026. What Is Codesign? [WWW Document]. IxDF - Interaction Design Foundation. URL https://ixdf.org/literature/topics/codesign (accessed 6.30.26). Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Vierling, L., Hong, D., Zhou, J., Zhang, Z., Zeng, F., Dai, J., Pan, X., Ng, K.Y., O...

arXiv 2025

[7] [2024]

Grand Challenge

The illusion of artificial inclusion. https://doi.org/10.1145/3613904.3642703 Argyle, L.P., Busby, E.C., Fulda, N., Gubler, J.R., Rytting, C., Wingate, D., 2023. Out of One, Ma ny: Using Language Models to Simulate Human Samples. Polit. Anal. 31, 337 –351. https://doi.org/10.1017/pan.2023.2 Bommasani, R., Creel, K.A., Kumar, A., Jurafsky, D., Liang, P., 2...

arXiv 2023

[8] [2025]

https://doi.org/10.48550/arXiv.2510.22954 Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A., 2024a

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond). https://doi.org/10.48550/arXiv.2510.22954 Kirk, H.R., Vidgen, B., Röttger, P., Hale, S.A., 2024a. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nat Mach Intell 6, 383 –392. https://doi.org/10.1038/s42256-024-00820-y Kir...

doi:10.48550/arxiv.2510.22954 2025