Pith. sign in

REVIEW 4 major objections 4 minor 5 references

Human-AI Synergy Supports Collective Creative Search

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hybrid human-AI groups achieve the highest semantic-search scores while preserving guess diversity, and both humans and AI agents adapt their strategies within the mix.

desk verdict Solid, important experiment: the hybrid human-AI performance-diversity result holds up, but the 'mutual adaptation' claim rests on a confounded diversity measure and the collective-peak headline overreaches. read the letter →

arxiv 2602.10001 v3 pith:7VOR2XPA submitted 2026-02-10 cs.SI cs.HC

Chenyi Li , Raja Marjieh , Haoyu Hu , Mark Steyvers , Katherine M. Collins , Ilia Sucholutsky , Nori Jacoby This is my paper · ORCID
classification cs.SIcs.HC
keywords collectivecreativityhuman-AIcollaborationsemanticsearchexploration-exploitationwordguessinggamediversitylargelanguagemodelshybridgroups
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies collective creativity as a controlled word-guessing game: players guess one word at a time, receive a similarity score from a word-embedding model, and see the best previous guess as a hint. It compares all-human, all-AI, and mixed human-AI groups. The central claim is that hybrid groups outperform both homogeneous groups on the highest similarity score reached, while keeping guess diversity statistically indistinguishable from human social groups. The paper further claims that the benefit is interactive: AI agents in hybrid groups produce more diverse and higher-scoring guesses than AI-only agents, and humans contribute more unique words when paired with AI. The authors interpret this as complementary exploration by humans and exploitation by AI, with both sides adapting to each other's presence.

What carries the argument

The game itself is the central mechanism: each round, a player makes ten single-word guesses scored by cosine similarity between the guess's word embedding and the hidden target's embedding; the best guess of the round becomes the hint for the next round, propagating through a linear chain. This setup converts open-ended creativity into a measurable search problem with a built-in exploration-exploitation tradeoff. The paper uses this to show that human guesses broaden the semantic territory while AI guesses concentrate it.

What would settle it

A replication using a different embedding model (for example, a transformer-based sentence encoder applied to the single words) that flips the ordering of conditions on peak score, or a version in which the hybrid benefit disappears when the AI agent is replaced by a pre-scripted list of guesses derived from human data, would falsify the claim that the interactive mix—rather than the metric or the information content—causes the advantage.

Watch

Extended reading notes

Core claim

On the study's own terms, the paper discovers that in a semantic search task with a hidden target and objective similarity feedback, hybrid human-AI groups reach higher peak scores than human-only or AI-only groups, and do so without the loss of diversity seen in AI-only groups. The effect is not additive: when placed in hybrid groups, AI agents improve in both performance and lexical diversity relative to AI-only conditions, and humans generate significantly more unique words. The paper presents this as evidence that collective creative advantage arises from dynamic interplay between complementary strategies, not from simply having more agents or more model variety.

Load-bearing premise

The paper's rankings of creativity and diversity rest on a specific geometric proxy: cosine similarity in a single Word2Vec embedding space, where both the score and the diversity measure are computed; if that space is not a faithful map of human-perceived semantic closeness, the condition differences could be an artifact of the metric. A second load-bearing premise is that the AI-only and hybrid-AI comparisons are tied to one particular language model and prompt, and the pap

Editorial extensions

If this is right

  • If hybrid groups truly dominate the performance-diversity frontier, then embedding AI assistants into collective search processes could raise group performance without the homogenization that individual AI assistance has been shown to cause.
  • AI agents' improvement in hybrid groups suggests that the value of an AI search agent depends on the social context it operates in; future AI systems may need to be designed to adapt to human exploration rather than simply maximize immediate similarity.
  • The failure of a mixed-AI group (two different models) to match human-AI performance implies that cognitive heterogeneity between humans and machines is qualitatively different from merely combining different machines.
  • The task's objective scoring scheme makes it a testbed for studying how information transmission rules (here, only the single best guess) shape collective search.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to test whether the hybrid advantage survives when the AI receives the full round history instead of only the best guess; the current design's minimal information channel may be a key part of the mechanism.
  • Because performance and diversity are both measured in the same embedding space, a different semantic model (or human ratings of novelty) could change the condition rankings; a replication along those lines would clarify whether the finding is about the embedding geometry or about the underlying search behavior.
  • The authors' interpretation suggests a design principle: in hybrid teams, the AI should be placed in a follower role that can rapidly exploit new regions opened by humans, rather than being the primary driver.
  • The 'uniqueness' effect on human behavior might extend to other forms of social influence—for instance, humans may generate more distinct ideas whenever they know a machine is part of the group, independent of the actual content of the AI's guesses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript studies collective creative search using a Semantle-style word-guessing game in which human participants and Gemini 2.5 Flash agents (in separate conditions and together) try to infer a hidden target word from cosine-similarity feedback and the previous best guess. Ten target words are used in 50 games; each game has 10 rounds of 10 guesses. The paper compares Human Social, Human Asocial, AI-only, and Human-AI Hybrid conditions on individual performance, collective peak score, and individual/collective diversity (1 - average pairwise cosine similarity). The main claims are: hybrid groups achieve the highest individual performance while preserving diversity comparable to human social groups; AI agents in hybrid groups improve; humans in hybrid groups contribute more unique words; and these benefits reflect complementary exploration-exploitation strategies, not mere agent diversity. Control experiments vary model type, prompt style, and social information.

Significance. The paper is a valuable contribution to the emerging literature on human-AI collective behavior. Its strengths are the controlled, interactive setup with real participants and contemporary LLMs; the inclusion of homogeneous and hybrid conditions; and multiple robustness controls (different LLM, prompts, social-information formats). The reported individual-performance advantage of hybrid groups over the other three conditions (p≤.026 after FDR) and the null diversity difference between Hybrid and Human Social (p=.536, d=0.04) are concrete and useful results, assuming the inference is valid. The main caveats are statistical: the collective peak-score claim is not tested, and the lexical-diversity evidence for mutual adaptation is confounded by unequal pooling sizes. These issues are fixable and do not invalidate the core experiment.

major comments (4)
  1. [Interaction with people changes AI behavior (Fig. 5B)] Lexical diversity is defined as the proportion of unique guessed words among all guesses pooled per hidden word. Because the number of guesses pooled differs sharply between conditions—Human Social has ~500 human guesses per target word while Hybrid has ~259; AI-only has ~500 AI guesses while Hybrid AI has ~241—the expected unique-proportion decreases mechanically with pool size. The paper reports no rarefaction or subsampling correction. Thus the p<.001 'human in Hybrid more unique' and 'AI in Hybrid more unique' results may be pure sample-size artifacts. This is load-bearing: it is the only significant human-side evidence for 'both humans and AI agents systematically adjust their strategies,' since the human performance gain is n.s. (p=.083) and individual diversity is n.s. Please re-analyze with fixed-size random subsamples (e.g., 241 or 259 guesses per target word) repeated over many
  2. [Performance analysis, collective peak score] The abstract and introduction state that hybrid groups achieve the highest performance, but the collective peak-score analysis is purely descriptive: Hybrid M=119.77 (CI [102.14,137.40]), Human Asocial 113.80, Human Social 113.16, AI-only 94.80. No p-value, effect size, or pairwise test is reported for this metric, and the 95% CIs overlap substantially between Hybrid and both human-only conditions. Either report a proper test with FDR correction or explicitly restrict the performance claim to the individual-performance measure in Fig. 3A.
  3. [Performance analysis / Fig. 3A and Fig. 5A] The reported t-tests use round-level degrees of freedom (e.g., t(757), t(739)), treating each round as an independent observation. In the Hybrid condition, 259 human rounds come from 28 participants (repeated measures); in Human Social, 500 rounds come from 50 participants; and rounds within a game share the same chain of hints. This clustering can inflate significance. Please add mixed-effects models with random intercepts for participant and game/target, or cluster-robust standard errors, for the main performance and adaptation comparisons.
  4. [Discussion / Methods construct validity] Both the performance score and the diversity measures are derived from the same Word2Vec embedding (cosine similarity to target; 1 - average pairwise cosine). The paper acknowledges the embedding-based proxy in the Discussion, but the abstract's claim about 'collective creativity' and the performance-diversity frontier would be substantially strengthened by a robustness analysis with at least one different embedding (e.g., GloVe, fastText) or a human-rated subset. Without this, the external validity of the central trade-off remains unclear.
minor comments (4)
  1. [Methods, Human Asocial] Methods state N=46 for Human Asocial but the design says 50 games total; clarify how 46 participants map onto games and target-word counterbalancing.
  2. [Fig. 6A] The Human-AI Hybrid point (M=83.52) is presented without a CI or a cross-reference to the corresponding condition in the main experiment; define it explicitly.
  3. [Results, collective performance] Typo: 'This metrics captures the frontier' should be 'This metric captures the frontier.'
  4. [General] No data/code availability statement is included; if the journal requires, please add one.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline performance/diversity comparisons are experimental observations, not derivations from fitted parameters or self-citations.

full rationale

The paper's central claims are empirical comparisons across four conditions. The scoring formula 'score=201.69∗cos(θ)' (Methods) contains an explicitly arbitrary constant that multiplies all scores, so it cannot manufacture a cross-condition ranking. No parameter is fitted to the data to produce the hybrid advantage; the 'mutual adaptation' narrative is supported by direct behavioral comparisons. The most plausible concern—lexical diversity defined as 'the proportion of unique guessed words among all the guesses across games with the same hidden word' (Results, Fig. 5B)—is a genuine statistical confound because Human Social and AI-only pool ~500 guesses per target word while Hybrid pools ~259 human and ~241 AI guesses; the proportion of unique types mechanically decreases with pool size. However, this is a sample-size artifact, not a circularity: the paper does not define the input in terms of the output, nor fit a parameter and rename it a prediction. Self-citations (Sucholutsky et al. 2025; Shiiku et al. 2025; Collins et al. 2024/2025) are motivational/programmatic and are not load-bearing for the headline result; no uniqueness theorem is imported. The Discussion explicitly limits the diversity measures as 'embedding-based proxies that may not fully capture novelty or usefulness' and conditions AI conclusions on 'the specific LLMs and prompting strategies used here,' which weakens external validity without making the derivation circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No free parameters are fitted to produce the headline comparisons; the 201.69 constant is an explicit arbitrary scaling that cancels in comparisons. The load-bearing choices are the embedding metric (Word2Vec), the specific LLM and prompt, and the linear chain structure — all acknowledged as limitations by the authors. No new entities are postulated; the 'humans explore, AI exploits' narrative is a post-hoc interpretive lens, not an entity with independent evidence, so it is scored under soundness and interpretation rather than as an invented entity.

free parameters (3)
  • 201.69 (score scaling constant) = 201.69
    Multiplicative constant in score = 201.69*cos(theta) (Methods). The authors call it 'an arbitrary number'; because it multiplies every score it cancels in all cross-condition comparisons and cannot drive reported differences. Listed for exhaustiveness.
  • UMAP hyperparameters = not reported
    Used only for the 2D visualization of trajectories (Fig. 2, S1). Not load-bearing for quantitative claims.
  • Target-word selection (10 words) = not applicable
    Words chosen 'to span a range of frequencies (and difficulty levels)'. Word choice affects absolute scores and difficulty, but condition comparisons are within the same word set, so this is a stimulus selection rather than a fitted parameter; it limits generalization to these 10 words.
assumptions (5)
  • domain assumption Word2Vec cosine similarity is a valid proxy for semantic closeness and creative search quality
    The entire scoring and diversity measurement depends on this embedding (Methods: 'score=201.69*cos(theta)'). No validation against human similarity judgments is provided; the authors acknowledge the diversity measures 'rely on embedding-based proxies that may not fully capture novelty or usefulness' (Discussion).
  • domain assumption LLM agents are valid participants comparable to humans given the same prompt and feedback
    AI agents received the same prompt ('Please enter your one-word guess') and feedback. LLM behavior is prompt-sensitive, and the authors' own control experiments (Fig. 6B) show large performance swings with prompt changes, so absolute AI-only baselines are a function of this specific prompt.
  • domain assumption The hidden maximum score (201.69) stays unlearned by participants, preserving exploration incentives
    Methods: 'we ensured that the maximum possible score (an arbitrary number, 201.69) was not revealed to participants, so that they would have an incentive to continue exploring.' If participants inferred the ceiling, dynamics could change.
  • standard math Rounds and games can be treated as roughly exchangeable observations for significance testing
    Reported tests use round-level degrees of freedom (e.g., t(757), t(739)) despite nesting of participants in games and target words; BH-FDR is applied, but the tests appear to be round-level rather than mixed models accounting for repeated measures.
  • domain assumption Not informing hybrid-condition humans that some partners are AI does not distort their behavior differentially
    Methods: 'participants were not informed that any guesses may have been generated by AI agents.' If humans detected AI-like guesses (e.g., the tightly clustered 'musical instrument' round in Fig. 2), their strategy adjustments could be a response to perceived AI presence rather than to the guesses themselves.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Human-AI Synergy Supports Collective Creative Search." pith.science (2026). https://pith.science/paper/7VOR2XPA

@misc{pith2026260210001,
  author       = {Pith},
  title        = {Pith review of: Human-AI Synergy Supports Collective Creative Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7VOR2XPA}},
  note         = {Machine review of arXiv:2602.10001}
}
read the original abstract

Generative AI is increasingly transforming creativity into a hybrid human-artificial process, but its impact on the quality and diversity of creative output remains unclear. We study collective creativity using a controlled word-guessing task that balances open-endedness with an objective measure of task performance. Participants attempt to infer a hidden target word, scored based on the semantic similarity of their guesses to the target, while also observing the best guess from previous players. We compare performance and outcome diversity across human-only, AI-only, and hybrid human-AI groups. Hybrid groups achieve the highest performance while preserving high diversity of guesses. Within hybrid groups, both humans and AI agents systematically adjust their strategies relative to single-agent conditions, suggesting higher-order interaction effects, whereby agents adapt to each other's presence. Although some performance benefits can be reproduced through collaboration between heterogeneous AI systems, human-AI collaboration remains superior, underscoring complementary roles in collective creativity.

Figures

Figures reproduced from arXiv: 2602.10001 by the authors.

Figure 1
Figure 1. Experiment framework for collective creative search. (A) Participants attempt to infer a hidden target word (“satellite”) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Semantic exploration trajectories. All words were [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance and diversity. A. Individual performance, computed as the average of the maximal score across rounds. Error bars represent one standard error across participants. Asterisks *, **, and *** denote significance levels of 0.05, 0.01, and 0.001, respectively. To account for multiple comparisons (here and in the rest of the paper), only results that passed the Benjamini–Hochberg False Discovery Rate (BH-FDR) c… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Collective performance-diversity relationship. Each [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Individual performance in control experiments. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 3 linked inside Pith

  1. [1]

    P., Paton, N., Watts, D

    Almaatouq, A., Becker, J., Houghton, J. P., Paton, N., Watts, D. J., & Whiting, M. E. (2021). Empirica: A virtual lab forhigh-throughputmacro-levelexperiments.BehaviorRe- search Methods,53(5), 2158–2171. Bellemare-Pepin, A., Lespinasse, F., Thölke, P., Harel, Y., Mathewson,K.,Olson,J.A.,Bengio,Y.,&Jerbi,K.(2026). Divergent creativity in humans and large l...

    2021

  2. [431]

    T., Todd, P

    Hills, T. T., Todd, P. M., Lazer, D., Redish, A. D., & Couzin, I.D.(2015).Explorationversusexploitationinspace,mind, and society.Trends in cognitive sciences,19(1), 46–54. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger,O.,Tunyasuvunakool,K.,Bates,R.,Žídek,A., Potapenko, A., et al. (2021). Highly accurate protein struc- turepredic...

    arXiv 2015

  3. [875]

    L., Henrich, J., et al

    Brinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller,T.F.,Nussberger,A.-M.,Czaplicka,A.,Acerbi,A., Griffiths, T. L., Henrich, J., et al. (2023). Machine culture. Nature Human Behaviour,7(11), 1855–1868. Collins,K.M.,Bhatt,U.,&Sucholutsky,I.(2025).Revisiting rogers’paradoxinthecontextofhuman-aiinteraction.arXiv preprint arXiv:2501.10476. Collins,...

    arXiv 2023

  4. [1279]

    R., Kruschke, J

    Breithaupt, F., Otenen, E., Wright, D. R., Kruschke, J. K., Li, Y., & Tan, Y. (2024). Humans create more novelty than chatgpt when asked to retell a story.Scientific Reports, 14(1),

    2024

  5. [1863]

    David Turner. (2022). Semantle: Daily word guessing game [Web-based semantic word game; players guess a hid- den word based on semantic similarity feedback].Seman- tle.com. Doshi, A. R., & Hauser, O. P. (2024). Generative ai enhances individual creativity but reduces the collective diversity of novel content.Science advances,10(28), eadn5290. Garg, K., Ke...

    arXiv 2022

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.