{"id":"0b583575-7eec-44f7-ba0a-328e29b7178d","arxiv_id":"2607.15545","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"CoWeaver combines capability-gap matching, UCB exploration, and LLM-based simulated negotiation to recommend collaborations in mixed human-agent networks, evaluated only on a synthetic benchmark.","lead":"CoWeaver is a matching engine that pairs scientists with collaborators or agents by filling capability gaps, checking hard constraints, and using LLM-simulated talks to rerank matches. It also explores cold-start candidates with an exploration-exploitation rule, but its only tests are on 20 synthetic tasks, so real-world impact is unproven.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ1's 'oracle condition' (priors = true means, zero noise) makes the reported 6/20 UCB advantage over greedy-only logically impossible; the protocol needs clarification or reproduction.","rationale":"The reader's CONDITIONAL verdict is reasonable, and I would keep it. However, the single most load-bearing concern is not primarily 'synthetic benchmark is unrepresentative'; it is that the RQ1 protocol as written makes the headline UCB advantage internally inconsistent. Section 3.2 says RQ1's oracle condition sets each prior mean to the true candidate mean (zero prior noise) with noise-free observations. Under that condition, greedy-only has exact capability estimates and, for the one-to-one selection evaluated by ρ_mode/Hit rate, is the optimal policy. UCB's bonus can only deviate from the greedy argmax; it cannot improve expected ρ_mode. The reported 6/20 wins therefore imply one of three unstated conditions: non-true priors, an asymmetric 'new candidate match score bonus,' or an objective mismatch between the Scap-greedy selector and θ=[0.4,-0.1]. The text's own numbers also conflict with Table B.3. This is a load-bearing internal-consistency problem, not a consensus disagreement: it questions whether the central empirical claim can occur as described. The proposed test — rerun RQ1 with σ_init=0 and no bonus — is decisive. If UCB then ties greedy exactly, the paper must disclose which added component produced the 6/20 result; if it does not tie, the protocol has an undisclosed source of asymmetry. I credit the paper for releasing code and detailed tables, which make this check feasible, and I am not accusing the authors of misreporting; the issue may be a missing ablation or imprecise wording. For these reasons the appropriate verdict remains CONDITIONAL (unchanged), with the condition being a corrected, self-consistent RQ1 comparison.","tokens_in":17412,"tokens_out":10961,"duration_ms":118578,"concrete_test":"Re-run RQ1 (Section 3.2) exactly, but set σ_init=0 and remove the \"new candidate match score bonus.\" Under zero prior noise with true prior means, UCB+Scap Greedy and Scap Greedy must produce identical selections and identical ρ_mode. If the 6/20 advantage persists, it is caused by the bonus or by mismatch between the selector objective and θ; if it disappears, the reported advantage depended on an asymmetric or undescribed condition. Also re-report Table B.3 values and reconcile them with the §3.2 text.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states RQ1 uses \"the oracle condition [which] sets each prior mean to the true candidate mean, corresponding to zero prior noise.\" With noise-free oracle observations (Section 3.1), greedy-only Scap Greedy has exact capability estimates and, for the one-to-one selection RQ1 evaluates, is the optimal one-step selector. UCB's exploration bonus (Section 2.5: M + c·σ̄_v·sqrt(log(t+1)/(n_v+1))) can only deviate from the greedy argmax; in expectation it cannot improve ρ_mode over greedy, and finite-sample 6/20 wins are not explained. The paper nevertheless reports UCB+Scap Greedy exceeds Scap Greedy on 6/20 tasks (Figure 2, P=0.049). This can happen only if (a) the prior means were not actually the true means (e.g., tier-derived μ_init from Section 3.1), (b) the \"new candidate match score bonus\" mentioned in §3.2 was applied to UCB but not to the greedy baseline, or (c) the evaluation weight θ=[0.4,-0.1] rewards a different objective than the capability-greedy selector. The text does not disambiguate, and its numbers conflict with Table B.3 (text: first-opt ≈11 and ρ_mode 0.983/0.953; table under tiered priors: 16.6±9.3 and 0.975/0.952). Thus the headline empirical claim lacks a coherent mechanism as described.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoWeaver, a matching engine for mixed human-agent scientific collaboration. It represents actors by open-ended capabilities, needs, and hard constraints; computes a bidirectional MapScore with an interpretable decomposition; applies a two-stage analytical/simulation pipeline (including LLM Dreaming); and maintains uncertainty-aware capability estimates with UCB-style exploration. The evaluation uses 20 synthetic tasks with 20 candidates each and reports four studies: online selection (RQ1), the effect of LLM Dreaming (RQ2), faithfulness of MapScore explanations (RQ3), and a comparison with AgenticPay-style and PeopleJoin baselines (RQ4). The central claims are that UCB+Greedy exceeds Greedy-only on 6/20 tasks and that CoWeaver outperforms baselines on all metrics.","tokens_in":17739,"tokens_out":9362,"duration_ms":102504,"significance":"If the claims held, the paper would contribute a modular, interpretable matching pipeline with a feedback loop and a modest exploration bonus over greedy selection. Strengths include the explicit code/data link, the detailed appendix with reproducibility settings, the separation of hard feasibility from utility ranking, and the honest limitations paragraph. However, the current evidence is not yet convincing: the headline RQ1 result is internally inconsistent with the oracle-protocol description and with Table B.3, the abstract overstates Table B.4, and the entire evaluation is synthetic and noise-free. The interpretability apparatus and the closed-form attribution are the most promising parts, but the central empirical claims need major revision before the paper can be recommended for publication.","major_comments":[{"comment":"The RQ1 'oracle condition' cannot support the headline UCB advantage. With priors set to true means and noise-free observations, Greedy-only has exact capability estimates; Select_t(v)=M+cσ̄_v√(log(t+1)/(n_v+1)) only adds a positive bonus and every deviation from the greedy argmax selects a candidate with lower true M. UCB+Greedy therefore cannot beat Greedy-only in expectation. A 6/20 win requires an unspecified mechanism (tier-derived priors, the 'new candidate match score bonus' mentioned in §3.2, or a different objective), none of which is defined or ablated. Moreover, the text's numbers (ρmode 0.983/0.953, first-opt ~11) disagree with Table B.3 (0.975/0.952, 16.6±9.3). The exact protocol and all implementation bonuses must be stated, or the claim withdrawn.","section":"§2.5/§3.2"},{"comment":"The abstract says COWEAVER 'outperforms baselines on all metrics,' but Table B.4 shows PeopleJoin reaches consensus faster (5.10±0.60 vs 6.5±4.3 rounds). The table itself acknowledges this ('PeopleJoin exhibits faster repeated-selection consensus'). Revise the claim to state where CoWeaver wins (quality, first-optimum) and where it does not. Also, the AgenticPay adaptation (one-by-one matching, 3 candidates) is a narrow baseline; this should be reflected in the conclusion.","section":"Abstract/§3.5, Table B.4"},{"comment":"The evaluation weight θ=[0.4,-0.1] gives negative weight to Sneed, whereas MapScore (Eq. in §2.3) and both selectors use positive Sneed. Unless Moptimum is explicitly a separate evaluation utility, 'greedy-only is the analytical best solution' is not well defined: the quantity UCB tries to recover is not the argmax of the selector's own score. Define Moptimum, state the selector's objective, and report sensitivity of ρmode to θ.","section":"§3.1, oracle-optimum definition"},{"comment":"The '6/20 tasks' claim is weak and internally inconsistent with the aggregate table. Table B.3 shows UCB+Greedy has lower mean ρmode than Greedy-only in both prior conditions (0.896 vs 0.982 uniform; 0.952 vs 0.975 tiered). The text claims 'exceeds Scap Greedy-only on 6/20 tasks' with a Mann-Whitney P=0.049; please report paired per-task differences, effect sizes, and confidence intervals, and explain how a 6/20 win count with lower aggregate means yields that P-value. Without this, the exploration benefit is not demonstrated.","section":"§3.1/§3.2, Table B.3"}],"minor_comments":[{"comment":"The formula for Sneed contains an extra comma ('min(ok_T , , nk_v)') and the quantity α·Q_k^T is not defined. Please fix the notation.","section":"Appendix B.1"},{"comment":"The phrase 'P-value = 0.049' should specify the exact Mann-Whitney procedure, the number of ties, and how the 'correction by sample size' was applied; the current wording is unclear.","section":"§3.2"},{"comment":"'Tmode' (deterministic time-match score) is not defined in Methods. Define it or cite where it is introduced.","section":"§3.3, Table 1"},{"comment":"The citation 'Lappas, Lukasz Liu, and Evimaria Terzi' contains an erroneous first name; correct to 'Lappas, Liu, and Terzi.'","section":"References"},{"comment":"The survey has n=16 and is appropriately labeled as motivation. The main text should not cite it as stronger evidence than that; consider adding an explicit sentence in §1 clarifying its motivational-only status.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript has a well-structured design and transparent appendices, but the headline empirical claim is currently not coherent. The RQ1 protocol must be clarified or reproduced with a clearly defined comparison; the text/Table B.3 discrepancy and the abstract/Table B.4 overclaim also need to be fixed. The synthetic, noise-free benchmark may be acceptable as a controlled study only if all claims are scoped accordingly. I recommend major revision rather than rejection because the structural components and reproducibility artifacts are promising, but the current central evidence is not sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CoWeaver is a well-put-together integration of capability-gap matching, hard feasibility gates, UCB exploration, LLM-based reranking, and closed-form explanation. That integration is the real contribution—the landscape table is honest, and no prior system covers all four dimensions. The explanation-faithfulness evaluation with perturbation oracles is better than what this area usually does, and they ship code and parameter tables.\n\nThe problem is the headline empirical claim. RQ1 says UCB+Scap Greedy exceeds Scap Greedy-only on 6/20 tasks, but as described, the oracle condition sets prior means to true means with zero noise. Under that condition, greedy with exact estimates is the optimal one-step selector; UCB's exploration bonus can only move away from the greedy argmax. The text then says the 6/20 result appears after adding 'tiered standard deviations and new candidate match score bonus'—but that bonus is never defined in the method. If it's applied to UCB but not the greedy baseline, the comparison is unfair. The numbers also conflict with Table B.3: text says first-opt ≈11 and ρmode 0.983/0.953; the tiered row shows 16.6±9.3 and 0.975/0.952, and the uniform row shows 11.0 and 0.982/0.896. None of the rows matches the text cleanly. The stress-test note's concern is right.\n\nOther soft spots: the abstract claims 'outperforms baselines on all metrics,' but Table B.4 shows PeopleJoin reaches consensus faster (5.10 vs 6.5), and Table 1 shows GREEDY beats GREEDY+LLM DREAMING on capability metrics. The evaluation is entirely synthetic with noise-free oracle observations, and candidates are generated centered around task skills—the limitations section admits this, but it still means the central results could be simulator artifacts. The free parameters λ, η, ρ, β_t, c, and the Ssim weights have no sensitivity analysis, and there's no commit hash for the code.\n\nWhat does hold up? The pipeline design is coherent, the closed-form attribution is a nice property, and the personality experiment using AgenticPay's compatibility matrix as held-out ground truth is a clever way to evaluate re-ranking. The work deserves a serious referee.\n\nWho is this for? Researchers building task-conditioned matching or team-formation systems, especially in human-agent settings. It's a useful systems paper, but not yet a validated empirical one.\n\nRecommendation: send it to peer review, with a clear signal that RQ1 needs rewriting: the protocol must specify whether the 6/20 result uses the oracle condition or the tiered/bonus variant, the bonus must be defined and given to both arms, and the abstract should be corrected. If the UCB advantage survives those changes, it's a meaningful result; if not, the paper still stands as a system contribution but the exploration claim should be dropped.","headline":"A genuinely integrated matching pipeline, but the headline UCB-vs-greedy result is not supported as written—text and Table B.3 conflict, and the 'new candidate match score bonus' is undefined.","tokens_in":18321,"tokens_out":6462,"would_cite":false,"duration_ms":64055,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoWeaver proposes that collaboration matching is a two-way, feasibility-gated fit, and shows that adding exploration to greedy selection beats the analytical best on 6 of 20 tasks.","keywords":["collaboration matching","human-agent networks","bidirectional matching","capability gap","exploration-exploitation","LLM agents","explainable matching","scientific collaboration"],"falsifier":"Rerun the RQ1 protocol on the same 20 tasks after adding zero-mean Gaussian noise to the oracle capability reports, with variance proportional to each candidate's uncertainty σ̄_v, and check whether UCB+greedy still beats greedy-only on 6/20 tasks; if the advantage does not survive noisy observations, the exploration claim is an artifact of the noise-free benchmark. As a second check, on the paper's 25-task personality set, compute the rank correlation between the LLM judge's Ssim and the outcome-grounded compatibility score — the reported Agreeable–Strategic divergence (0.74 vs 0.03) predicts","tokens_in":17185,"feed_emoji":"🤝","tokens_out":12835,"duration_ms":123328,"temperature":0.7,"pith_summary":"CoWeaver sets out to show that forming collaborations in mixed human-agent science networks is a task-conditioned, two-way matching problem: a candidate must fill the requester's specific capability gaps, and the task must satisfy the candidate's own needs, all under hard feasibility constraints. The paper builds a score, MapScore, that computes this bidirectional fit behind a feasibility gate, then wraps it in a two-stage pipeline — fast analytical ranking followed by LLM-simulated negotiation for top candidates — and a feedback loop that explores under-observed newcomers with an uncertainty bonus. It claims this combined mechanism beats greedy-only selection, which it identifies as the analytical best, on 6 of 20 tasks and matches it on best-candidate recovery, while the full system outperforms an adapted negotiation baseline and a multi-user coordination baseline on matching quality and efficiency. A reader should care because, if true, this offers an interpretable, principled way to recommend collaborators in settings where humans and AI agents coexist and where researchers often know the bottleneck but not the right partner.","feed_headline":"Exploration bonus beats pure greedy matching on 6 of 20 tasks","feed_subtitle":"CoWeaver fills capability gaps on both sides of a match; an uncertainty bonus finds near-optimal collaborators fast.","key_machinery":"The central object is MapScore, a feasibility-gated, bidirectional matching score: M(u,v,T) = σ(u,v,T)[λ Scap(u,v,T) + (1−λ) Sneed(v,T)], where σ is a hard gate over non-negotiable constraints, Scap is the fraction of the requester's residual capability gap that the candidate's optimistic coverage c_j^v = min(1, p̃_j^v + β_t σ̃_j^v) can fill, and Sneed is the weighted fraction of the candidate's needs the task's offers satisfy. Because both terms are closed-form over atomic requirement and need factors, the score doubles as an explanation layer. Three mechanisms use it: a residual-gap greedy loop that builds teams by marginal coverage, a UCB exploration bonus Select_t(v) = M + c σ̄_v sqrt(lo","core_discovery":"The central claim is that collaboration value decomposes into two asymmetric questions — whether a candidate covers the requester's residual capability gap (Scap) and whether the task satisfies the candidate's participation needs (Sneed) — combined behind a hard feasibility gate σ. CoWeaver instantiates this in MapScore, M = σ[λ Scap + (1−λ) Sneed], and shows that a greedy selector on this score recovers oracle-optimal collaborators under a 25-round interaction budget. When the selector adds an upper-confidence exploration bonus for under-observed candidates, it beats greedy-only on 6 of 20 tasks and matches it on best-candidate recovery; with tiered prior uncertainty, quality stays close to","pith_inferences":["The 6/20 UCB-over-greedy advantage is measured on noise-free oracle observations; rerunning the protocol with noisy capability reports (variance growing with uncertainty) would show whether the advantage is a benchmark artifact or a general property.","The paper's Agreeable–Strategic failure — an LLM judge rates a lopsided deal as compatible (Ssim 0.74) while an outcome-grounded fairness score is 0.03 — implies that simulation-based compatibility judgments need an explicit fairness check on the final allocation; the gap is structural, not a prompt-level fix.","The closed-form attribution could power a post-hoc audit: record which factors were shown before each collaboration and regress actual outcomes on those same factors to test whether the stated reasons were predictive, extending the paper's λ-update into a fuller prediction layer."],"forward_implications":["A capability-gap-filling score alone can recover oracle-optimal collaborators under a limited interaction budget; the UCB exploration bonus beats pure greedy on 6 of the 20 tasks and matches it on best-candidate recovery.","LLM Dreaming reranking improves collaboration score, time feasibility, and combined utility at a small capability-optimality cost, so it functions as a feasibility-correction layer rather than a replacement for the analytical score.","The full CoWeaver system exceeds the adapted AgenticPay-style and PeopleJoin baselines on both quality and efficiency in the tested protocol (ρmode 0.977 vs 0.941/0.966; hit rate 0.75 vs 0.25/0.333; rounds to ρ≥0.99 2.06 vs 20.0/5.21).","MapScore's top-attributed explanation factor aligns with the strongest single-factor oracle intervention on the contrast benchmark (ORat=1.0, OMatch=1.0), supporting the claim that the displayed reasons reflect the actual decision.","Uncertainty-aware exploration with tiered priors keeps matching quality close to pure greedy (ρmode 0.953 vs 0.983), so under-tested candidates can enter the shortlist without wrecking recommendation quality in the tested protocol."],"fun_headline_variants":["Exploration boost tops greedy matching on 6 of 20 tasks","CoWeaver: UCB beats greedy on 6 of 20 collaboration tasks","Capability-gap matching with uncertainty bonus wins on 6 tasks","Mixed-agent matching: explore to beat greedy on 6 of 20","CoWeaver beats greedy when exploration finds best collaborators"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the synthetic 20-task benchmark, with noise-free oracle capability observations and with 'best candidate' defined by the authors' own evaluation weight θ=[0.4,−0.1], is representative of real mixed human-agent collaboration; if real deployments involve noisy evidence, strategic behavior, and long-term dynamics (the paper's own limitations list), the measured 6/20 UCB advantage and the baseline superiority may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Exploration boost tops greedy matching on 6 of 20 tasks","CoWeaver: UCB beats greedy on 6 of 20 collaboration tasks","Capability-gap matching with uncertainty bonus wins on 6 tasks","Mixed-agent matching: explore to beat greedy on 6 of 20","CoWeaver beats greedy when exploration finds best collaborators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000652,"raw_usage":{"total_tokens":2815,"prompt_tokens":722,"completion_tokens":2093,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":2001}},"tokens_in":466,"tokens_out":2093,"duration_ms":14120,"temperature":1.0,"reasoning_tokens":2001,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:59:41.023882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the RQ1 protocol on the same 20 tasks after adding zero-mean Gaussian noise to the oracle capability reports, with variance proportional to each candidate's uncertainty σ̄_v, and check whether UCB+greedy still beats greedy-only on 6/20 tasks; if the advantage does not survive noisy observations, the exploration claim is an artifact of the noise-free benchmark. As a second check, on the paper's 25-task personality set, compute the rank correlation between the LLM judge's Ssim and the outcome-grounded compatibility score — the reported Agreeable–Strategic divergence (0.74 vs 0.03) predicts","supporting_citations":[],"review_version":1}