{"id":"5d84ccc9-cf9d-42dd-9016-b28e4afe644b","arxiv_id":"2602.10001","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Hybrid human-AI groups outperform all-human and all-AI groups on a semantic word-search task while keeping guess diversity as high as all-human groups.","lead":"Researchers ran a word-guessing game, like Semantle, in which humans, AI agents, or both took turns trying to find a hidden word, and compared how well each group performed. They found that human-AI teams reached better words fastest while keeping their guesses diverse, and each side changed its behavior when paired with the other.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lexical-diversity measure confounded by unequal guess counts; human adaptation evidence may be artifactual.","rationale":"The reader's weakest_assumption focuses on the Word2Vec metric as a proxy for creativity and the prompt/model specificity of the AI baseline. These are valid external-validity concerns. However, I identify a more immediate, internal-validity problem in a specific piece of evidence used to support the 'mutual adaptation' claim: the lexical diversity measure is confounded by unequal numbers of guesses between conditions. This does not overturn the core finding that Hybrid achieved the highest individual performance and preserved diversity—those analyses use different metrics and are statistically supported. But it directly undermines the statement that 'both humans and AI agents systematically adjust their strategies' because the only significant human-side adjustment (unique-word proportion) is confounded. The verdict should remain CONDITIONAL, as the central performance claim is credible but the mechanistic interpretation needs additional support. My check would decide whether this concern actually lands; if rarefaction removes the effect, the human-adaptation portion of the abstract's claim should be softened.","tokens_in":10684,"tokens_out":6643,"duration_ms":73501,"concrete_test":"Recompute lexical diversity after rarefaction: for each target word and condition, randomly subsample the same number of guesses (e.g., the minimum across conditions, ~241 for AI and ~259 for humans) without replacement, repeat 1000 times, and compare the mean unique-word proportion between conditions. If the p<.001 differences disappear or shrink below significance, the reported lexical-diversity advantage is a sampling artifact. Additionally, report unique words as a function of cumulative guesses (accumulation curves) to show whether the Hybrid advantage persists at matched sample sizes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's mechanism claim ('both humans and AI agents systematically adjust their strategies') rests partly on the finding that humans in the Hybrid condition contributed a significantly higher proportion of unique words than in Human Social (p<.001), and AI in Hybrid higher than AI-only (p<.001). However, 'lexical diversity' is defined as the proportion of unique guessed words among all guesses pooled across games with the same hidden word. The number of pooled guesses per target word differs sharply between conditions: Human Social has ~500 human guesses per word, while Hybrid has only ~259 human guesses per word; likewise AI-only has ~500 AI guesses vs ~241 in Hybrid. The expected proportion of unique types mechanically decreases as sample size increases (more guesses yield more repeats), so the Hybrid condition's higher uniqueness may be purely an artifact of having fewer total guesses, not a behavioral change. The paper does not mention rarefaction or any sample-size correction. If this confound holds, the human-specific adaptation claim loses its only significant evidence (human performance gain was p=.083), weakening the 'both agents adapt' central narrative, even though the main performance-diversity result could remain intact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies collective creative search using a Semantle-style word-guessing game in which human participants and Gemini 2.5 Flash agents (in separate conditions and together) try to infer a hidden target word from cosine-similarity feedback and the previous best guess. Ten target words are used in 50 games; each game has 10 rounds of 10 guesses. The paper compares Human Social, Human Asocial, AI-only, and Human-AI Hybrid conditions on individual performance, collective peak score, and individual/collective diversity (1 - average pairwise cosine similarity). The main claims are: hybrid groups achieve the highest individual performance while preserving diversity comparable to human social groups; AI agents in hybrid groups improve; humans in hybrid groups contribute more unique words; and these benefits reflect complementary exploration-exploitation strategies, not mere agent diversity. Control experiments vary model type, prompt style, and social information.","tokens_in":10803,"tokens_out":9438,"duration_ms":97752,"significance":"The paper is a valuable contribution to the emerging literature on human-AI collective behavior. Its strengths are the controlled, interactive setup with real participants and contemporary LLMs; the inclusion of homogeneous and hybrid conditions; and multiple robustness controls (different LLM, prompts, social-information formats). The reported individual-performance advantage of hybrid groups over the other three conditions (p≤.026 after FDR) and the null diversity difference between Hybrid and Human Social (p=.536, d=0.04) are concrete and useful results, assuming the inference is valid. The main caveats are statistical: the collective peak-score claim is not tested, and the lexical-diversity evidence for mutual adaptation is confounded by unequal pooling sizes. These issues are fixable and do not invalidate the core experiment.","major_comments":[{"comment":"Lexical diversity is defined as the proportion of unique guessed words among all guesses pooled per hidden word. Because the number of guesses pooled differs sharply between conditions—Human Social has ~500 human guesses per target word while Hybrid has ~259; AI-only has ~500 AI guesses while Hybrid AI has ~241—the expected unique-proportion decreases mechanically with pool size. The paper reports no rarefaction or subsampling correction. Thus the p<.001 'human in Hybrid more unique' and 'AI in Hybrid more unique' results may be pure sample-size artifacts. This is load-bearing: it is the only significant human-side evidence for 'both humans and AI agents systematically adjust their strategies,' since the human performance gain is n.s. (p=.083) and individual diversity is n.s. Please re-analyze with fixed-size random subsamples (e.g., 241 or 259 guesses per target word) repeated over many","section":"Interaction with people changes AI behavior (Fig. 5B)"},{"comment":"The abstract and introduction state that hybrid groups achieve the highest performance, but the collective peak-score analysis is purely descriptive: Hybrid M=119.77 (CI [102.14,137.40]), Human Asocial 113.80, Human Social 113.16, AI-only 94.80. No p-value, effect size, or pairwise test is reported for this metric, and the 95% CIs overlap substantially between Hybrid and both human-only conditions. Either report a proper test with FDR correction or explicitly restrict the performance claim to the individual-performance measure in Fig. 3A.","section":"Performance analysis, collective peak score"},{"comment":"The reported t-tests use round-level degrees of freedom (e.g., t(757), t(739)), treating each round as an independent observation. In the Hybrid condition, 259 human rounds come from 28 participants (repeated measures); in Human Social, 500 rounds come from 50 participants; and rounds within a game share the same chain of hints. This clustering can inflate significance. Please add mixed-effects models with random intercepts for participant and game/target, or cluster-robust standard errors, for the main performance and adaptation comparisons.","section":"Performance analysis / Fig. 3A and Fig. 5A"},{"comment":"Both the performance score and the diversity measures are derived from the same Word2Vec embedding (cosine similarity to target; 1 - average pairwise cosine). The paper acknowledges the embedding-based proxy in the Discussion, but the abstract's claim about 'collective creativity' and the performance-diversity frontier would be substantially strengthened by a robustness analysis with at least one different embedding (e.g., GloVe, fastText) or a human-rated subset. Without this, the external validity of the central trade-off remains unclear.","section":"Discussion / Methods construct validity"}],"minor_comments":[{"comment":"Methods state N=46 for Human Asocial but the design says 50 games total; clarify how 46 participants map onto games and target-word counterbalancing.","section":"Methods, Human Asocial"},{"comment":"The Human-AI Hybrid point (M=83.52) is presented without a CI or a cross-reference to the corresponding condition in the main experiment; define it explicitly.","section":"Fig. 6A"},{"comment":"Typo: 'This metrics captures the frontier' should be 'This metric captures the frontier.'","section":"Results, collective performance"},{"comment":"No data/code availability statement is included; if the journal requires, please add one.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has an interesting design and the core individual-performance result is worth publishing if the statistical issues are addressed. The lexical-diversity confound is fixable with rarefaction, but it directly affects the abstract's mutual-adaptation claim, so it cannot be treated as a minor wording issue. I recommend major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You can read this paper as the natural next step after Ueshima et al. and Shiiku et al.: real LLM agents embedded in semantic search chains, compared against all-human and all-AI chains. The core result is a controlled demonstration that hybrid human-AI chains beat homogeneous chains on individual peak similarity while keeping diversity statistically indistinguishable from human social groups. That's a real finding, FDR-corrected, and the authors are appropriately cautious about the single-embedding creativity proxy.\n\nWhat's genuinely new is the attempt to measure second-order effects: AI agents perform better in hybrid than alone, and humans show a modest bump. The control experiments with two LLMs and with different prompts are thoughtful and honestly reported. The main design is clean, and the scaling constant is arbitrary but multiplies all scores equally, so no fitting is happening.\n\nThe biggest soft spot is the lexical diversity measure. It's defined as the proportion of unique guessed words among all guesses pooled per target word. Pool size differs sharply by condition: human social has roughly twice as many human guesses per word as hybrid. No rarefaction or sample-size correction appears. So the p<.001 claim that 'humans contribute more unique words in hybrid' is likely an artifact of smaller sample size, not a behavioral change. That matters because the abstract's 'both humans and AI systematically adjust their strategies' hangs on this for humans—their performance gain was not significant (p=.083). The AI performance gain is computed from per-round maximum scores, so it isn't confounded the same way; that part stands. But the mutual-adaptation narrative is weakened.\n\nThe collective peak-score headline is also overclaimed: Hybrid's mean is highest but the 95% CIs overlap the human conditions and no significance test is reported. The negative performance-diversity correlation may be partly mechanical since both quantities come from the same Word2Vec geometry, though the authors do acknowledge the metric's limits. Minor: no code or data shipped, and the AI results are prompt- and model-specific, though they tested several variations.\n\nThis paper is for anyone working on collective intelligence, human-AI teams, or the AI-homogenization debate. I'd bring it to reading group and would cite the main result in my own work. It deserves a serious referee: the experiment is well-designed and the core finding is likely real, but the authors should be asked to fix the lexical diversity analysis—rarefaction or per-participant mixed models—and to moderate the 'both adapt' claim unless it survives that fix.","headline":"Solid, important experiment: the hybrid human-AI performance-diversity result holds up, but the 'mutual adaptation' claim rests on a confounded diversity measure and the collective-peak headline overreaches.","tokens_in":11468,"tokens_out":2557,"would_cite":true,"duration_ms":27368,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid human-AI groups achieve the highest semantic-search scores while preserving guess diversity, and both humans and AI agents adapt their strategies within the mix.","keywords":["collective creativity","human-AI collaboration","semantic search","exploration-exploitation","word guessing game","diversity","large language models","hybrid groups"],"falsifier":"A replication using a different embedding model (for example, a transformer-based sentence encoder applied to the single words) that flips the ordering of conditions on peak score, or a version in which the hybrid benefit disappears when the AI agent is replaced by a pre-scripted list of guesses derived from human data, would falsify the claim that the interactive mix—rather than the metric or the information content—causes the advantage.","tokens_in":10441,"feed_emoji":"🤝","tokens_out":4313,"duration_ms":43776,"temperature":0.7,"pith_summary":"The paper studies collective creativity as a controlled word-guessing game: players guess one word at a time, receive a similarity score from a word-embedding model, and see the best previous guess as a hint. It compares all-human, all-AI, and mixed human-AI groups. The central claim is that hybrid groups outperform both homogeneous groups on the highest similarity score reached, while keeping guess diversity statistically indistinguishable from human social groups. The paper further claims that the benefit is interactive: AI agents in hybrid groups produce more diverse and higher-scoring guesses than AI-only agents, and humans contribute more unique words when paired with AI. The authors interpret this as complementary exploration by humans and exploitation by AI, with both sides adapting to each other's presence.","feed_headline":"Human-AI hybrid groups outperform all-human and AI-only teams","feed_subtitle":"In a controlled guessing game, mixed teams reached higher scores without sacrificing guess diversity.","key_machinery":"The game itself is the central mechanism: each round, a player makes ten single-word guesses scored by cosine similarity between the guess's word embedding and the hidden target's embedding; the best guess of the round becomes the hint for the next round, propagating through a linear chain. This setup converts open-ended creativity into a measurable search problem with a built-in exploration-exploitation tradeoff. The paper uses this to show that human guesses broaden the semantic territory while AI guesses concentrate it.","core_discovery":"On the study's own terms, the paper discovers that in a semantic search task with a hidden target and objective similarity feedback, hybrid human-AI groups reach higher peak scores than human-only or AI-only groups, and do so without the loss of diversity seen in AI-only groups. The effect is not additive: when placed in hybrid groups, AI agents improve in both performance and lexical diversity relative to AI-only conditions, and humans generate significantly more unique words. The paper presents this as evidence that collective creative advantage arises from dynamic interplay between complementary strategies, not from simply having more agents or more model variety.","pith_inferences":["A natural extension is to test whether the hybrid advantage survives when the AI receives the full round history instead of only the best guess; the current design's minimal information channel may be a key part of the mechanism.","Because performance and diversity are both measured in the same embedding space, a different semantic model (or human ratings of novelty) could change the condition rankings; a replication along those lines would clarify whether the finding is about the embedding geometry or about the underlying search behavior.","The authors' interpretation suggests a design principle: in hybrid teams, the AI should be placed in a follower role that can rapidly exploit new regions opened by humans, rather than being the primary driver.","The 'uniqueness' effect on human behavior might extend to other forms of social influence—for instance, humans may generate more distinct ideas whenever they know a machine is part of the group, independent of the actual content of the AI's guesses."],"forward_implications":["If hybrid groups truly dominate the performance-diversity frontier, then embedding AI assistants into collective search processes could raise group performance without the homogenization that individual AI assistance has been shown to cause.","AI agents' improvement in hybrid groups suggests that the value of an AI search agent depends on the social context it operates in; future AI systems may need to be designed to adapt to human exploration rather than simply maximize immediate similarity.","The failure of a mixed-AI group (two different models) to match human-AI performance implies that cognitive heterogeneity between humans and machines is qualitatively different from merely combining different machines.","The task's objective scoring scheme makes it a testbed for studying how information transmission rules (here, only the single best guess) shape collective search."],"fun_headline_variants":["Mixed human-AI groups outperform without diversity loss","Human-AI collaboration yields top creativity, keeps diversity","AI and humans adapt to each other, lifting group creativity","Hybrid teams beat all-human and AI-only in creative search","Complementary not additive: Human-AI synergy boosts creativity"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's rankings of creativity and diversity rest on a specific geometric proxy: cosine similarity in a single Word2Vec embedding space, where both the score and the diversity measure are computed; if that space is not a faithful map of human-perceived semantic closeness, the condition differences could be an artifact of the metric. A second load-bearing premise is that the AI-only and hybrid-AI comparisons are tied to one particular language model and prompt, and the pap","fun_headline_variants_meta":{"raw":{"variants":["Mixed human-AI groups outperform without diversity loss","Human-AI collaboration yields top creativity, keeps diversity","AI and humans adapt to each other, lifting group creativity","Hybrid teams beat all-human and AI-only in creative search","Complementary not additive: Human-AI synergy boosts creativity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001125,"raw_usage":{"total_tokens":4466,"prompt_tokens":648,"completion_tokens":3818,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":392,"completion_tokens_details":{"reasoning_tokens":3739}},"tokens_in":392,"tokens_out":3818,"duration_ms":24925,"temperature":1.0,"reasoning_tokens":3739,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T02:36:31.138480+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A replication using a different embedding model (for example, a transformer-based sentence encoder applied to the single words) that flips the ordering of conditions on peak score, or a version in which the hybrid benefit disappears when the AI agent is replaced by a pre-scripted list of guesses derived from human data, would falsify the claim that the interactive mix—rather than the metric or the information content—causes the advantage.","supporting_citations":[],"review_version":1}