{"id":"c874bcf6-9b68-490c-b737-1374dda90851","arxiv_id":"2509.05764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"DRF combines an LLM rating network, a reputation update rule, and a UCB-style selection strategy to filter low-quality LLM agents during multi-agent task execution, reporting improved pass@1 and lower simulated cost on HumanEval and BigBench.","lead":"DRF is a framework for LLM multi-agent teams that scores each agent's reputation, updates it after every task round, and uses a bandit-style rule to pick the best agents. The paper reports quality and cost gains over several baselines on code and logical reasoning benchmarks, though the cost numbers are simulated and no code is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (9) adds cost with a positive sign in the UCB score, so the printed selection rule prefers expensive agents; the reported cost savings in Tables 3-4 cannot be produced as stated.","rationale":"The reader identified evaluator trustworthiness as the weakest assumption; that is a valid external threat, but it depends on unknown LLM behavior. The Eq. (9) sign error is more decisive because it is an internal, deterministic contradiction: the selection rule as written maximizes cost while the objective and the reported efficiency results claim cost minimization. This directly undermines the collaboration-efficiency component of the central claim and makes the experimental results non-reproducible from the stated method. In good faith, this is likely a typographical/sign error rather than fraud; after correction and rerunning, the empirical claim could still hold. Therefore I do not move away from the reader's CONDITIONAL verdict, but I would add an explicit condition: correct Eq. (9) (and check Eq. (2)) and show that the cost results survive. This reinforces the conditional verdict rather than overturning it.","tokens_in":804,"tokens_out":885,"duration_ms":132041,"concrete_test":"Implement Algorithm 2 exactly as printed, including Eq. (9), on a small DRF configuration (e.g., 6 agents, 30 tasks) with the paper's cost model and delta=0.7; record the mean cost of actually selected agents. Then repeat with the sign-corrected rule S_i^t = delta * r_i^(t-1) - (1-delta) * c_i^t + x_i^(t-1). If the printed rule selects higher-cost agents than the corrected rule, the cost portion of Tables 3-4 cannot be attributed to the published algorithm; if the two are indistinguishable, the cost claim is not sensitive to the sign and should be re-derived.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DRF's stated objective is to minimize cost: Definition 4 defines C_i^t as an agent's bid, Eq. (4) minimizes average cost, and Section 5.1 draws costs from U(0,1) with lower values intended to be better. But Eq. (9) computes S_i^t = delta * r_i^(t-1) + (1-delta) * c_i^t + x_i^(t-1), adding cost with a positive coefficient. In the comparative setting (Section 5.3) delta=0.7, so the rule as written systematically favors higher-cost agents. With the stated independent U(0,1) costs, a literal implementation would select agents whose expected cost exceeds the random baseline, while the paper claims DRF's average cost is lower than comparison frameworks. The only reconciliations are that the implementation used a different sign (e.g., -(1-delta)*c_i or (1-delta)*(1-c_i)) or that reported costs were generated by a different selection process; either way, the published algorithm is not what produced Tables 3-4. This is a load-bearing internal inconsistency for the collaboration-efficiency half of the central claim. Equation (2) shows the same pattern: A_i^t = abs(w_0^t - w_i^t)/w_0^t increases as the agent score moves away from the threshold, inverting the intended accuracy semantics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DRF, a framework for LLM-based multi-agent systems that combines a k-layer rating network, a reputation iteration mechanism, and a UCB-based selection strategy. The intended contribution is to dynamically filter high-reputation/low-cost agents and eliminate underperforming or malicious agents. Experiments on HumanEval and BigBench logical reasoning compare DRF with DyLAN, Reflexion, CodeT, and LLM-Debate and report improved pass@1/accuracy and lower cost.","tokens_in":10258,"tokens_out":4844,"duration_ms":49975,"significance":"The idea is timely and the combination of reputation scoring with UCB exploration is reasonable; if shown valid, it could give practical gains in LLM multi-agent collaboration. The paper has a clear system model and makes falsifiable benchmark claims. However, the current version contains several internal contradictions in the formalization and the experimental evidence does not substantiate the headline claims. The core mechanism is plausible but not yet supported.","major_comments":[{"comment":"Equation (9) defines S_i^t = δ r_i^{t-1} + (1-δ)c_i^t + x_i^{t-1}. Since Definition 4 and Eq. (1) treat cost as something to minimize, and §5.3 sets δ=0.7 with costs drawn from U(0,1), this selection rule rewards higher cost. The reported cost reductions in Tables 3-4 therefore cannot be a consequence of the published algorithm. Either the sign should be −(1−δ)c_i^t / (1−c_i^t), or the implementation differed; in both cases the reported experiments need to be rerun and the description corrected.","section":"§4.3, Eq. (9)"},{"comment":"Equation (2) defines A_i^t = |w_0^t − w_i^t| / w_0^t. For an agent whose score exactly meets the threshold, accuracy is 0, while an agent far from the threshold gets high accuracy. This is the opposite of Definition 3 ('higher accuracy indicates a smaller discrepancy') and makes the maximization in Eq. (4) select poor agents. If the intended quantity is distance, the objective must be minimization; otherwise the accuracy should be 1 − distance.","section":"§3.2, Eq. (2)"},{"comment":"The decay step r_i^t = r_i^{t-1} − w_i^t r_i^{t-1} β implies that smaller w_i^t yields smaller absolute penalty (and w_i^t=0 gives no decay), yet the text says 'the lower the task score, the greater the decline in reputation.' This is numerically wrong and weakens the claimed ability to identify low-capability agents. The update should depend on the gap below the threshold, e.g., (w_0 − w_i^t).","section":"§4.2, Eq. (8)"},{"comment":"The cost figures are generated from a uniform distribution rather than measured (§5.1), only 30 HumanEval cases are sampled, and Tables 3–4 report point averages with no standard deviations, confidence intervals, or significance tests. Thus the central claims of 'significantly improves task completion quality and collaboration efficiency' are not statistically supported. The comparison also fixes δ=1 in Figure 3 but δ=0.7 in Table 3 without discussing the effect of this hyperparameter.","section":"§5.1–5.3, Tables 3–4"},{"comment":"The reputation signal is entirely internal: w_i^t is a reputation-weighted average of LLM ratings, and those ratings are weighted by the very reputations they update. The only external anchor in the experiments is the synthetic low/medium/high label, which is also handed to the agents in the Executor prompt (Table 2). This makes the detection results partly a test of the LLM's willingness to follow an explicit label, not of the framework's ability to discover unknown capability. An experiment with hidden capabilities, and ideally with colluding/malicious raters, is needed to support the framework's stated purpose.","section":"§4.1–4.2, Table 2"}],"minor_comments":[{"comment":"w_0^t is not defined precisely; it is only called 'the score threshold within the same paper.'","section":"§3.2, Eq. (2)"},{"comment":"Equation (5) uses ∅ without explanation; the budget constraint is unclear.","section":"§3.2, Eq. (5)"},{"comment":"R0 is called 'the threshold of reputation' in the text but 'the upper bound of reputation' in §4.3; these are contradictory.","section":"§4.3, Algorithm 2"},{"comment":"Algorithm 2's signature lists σ but σ is never used.","section":"§4.3, Algorithm 2"},{"comment":"Reference [20] is cited for CodeT in Table 3, but the reference list entry [20] is a crowdsourcing truth-inference paper, not CodeT; the citation appears to be mismatched.","section":"Table 3"},{"comment":"Table 4 cites LLM-Debate as [14], but [14] is BigBench; the debate method's original reference is missing.","section":"Table 4"},{"comment":"Symbol conflict: δ is both the reputation increment coefficient (Eq. 7) and the reputation-cost weight (Eq. 9).","section":"§4.2 and §4.3"},{"comment":"Algorithm 1 uses w_o while the text uses w_0; please use consistent notation.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript may be salvageable, but the current version has enough internal contradictions that the reported numbers cannot be taken at face value. The authors should correct the sign errors and rerun the experiments; without access to code or raw results, I cannot verify whether the reported tables correspond to any consistent implementation. I would also encourage the editor to ask for statistics (error bars / significance) and a hidden-ability setting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the DRF paper. The core combination — DyLAN-style mutual evaluation, a reputation increment/decay update, and UCB selection — is a reasonable modular extension, and the authors are honest about their debts to DyLAN and to their own MAB-RP work. The reputation-weighted rating network is a clean way to aggregate peer evaluations, and the synthetic low/medium/high capability setup is a sensible test bed. That is the good part.\n\nThe stress-test note lands, and it lands hard. Equation (9) adds cost with a positive sign: S_i^t = δr_i^{t-1} + (1-δ)c_i^t + x_i^{t-1}. Since lower cost is supposed to be better (Section 5.1, U(0,1) costs), the rule as printed prefers expensive agents. With δ=0.7, a literal implementation would select high-cost agents on average, so the reported cost savings in Tables 3-4 cannot be produced by this algorithm. Either the sign is a typo and the experiments used a different rule, or the published algorithm is not the one that ran. That breaks the collaboration-efficiency half of the central claim.\n\nEquation (2) is similarly inverted: A_i^t = |w_0 - w_i|/w_0. This is a distance from threshold, so an agent scoring exactly at the threshold gets zero accuracy, and a worse agent scores higher. The paper's objective is to maximize average accuracy, which would reward the worst agents. Another load-bearing sign error.\n\nThe experiments are weak: no error bars or significance tests, 30 HumanEval cases, hand-picked hyperparameters, and costs simulated rather than measured. The paper honestly discloses the simulation, but it doesn't fix the sign problem. The reader's conditional verdict is generous; the internal contradictions make the core results unreproducible as stated.\n\nOn the positive side, the writing is honest about prior work, and the self-referential reputation loop is a genuine modeling choice rather than an oversight. But the paper needs a major correction before it can be taken seriously.\n\nRecommendation: do not send this to peer review as is. If the authors fix Eqs. (2) and (9) and rerun with real cost accounting and statistics, it might be worth another look. I wouldn't cite it in its current state, though I'd flag it as a cautionary example for a reading group.","headline":"The DRF framework is a plausible DyLAN extension, but Eq. (9)'s positive cost sign inverts the cost objective, so the reported cost savings cannot be reproduced as written.","tokens_in":10772,"tokens_out":4051,"would_cite":false,"duration_ms":42791,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dynamic reputation filter that lets LLM agents rate each other and then uses a bandit rule to pick the next team improves task quality and cuts cost on code and reasoning benchmarks.","keywords":["LLM multi-agent systems","reputation","multi-armed bandit","dynamic team selection","peer evaluation","code generation","logical reasoning","LLM-as-judge"],"falsifier":"Run the same benchmarks with a team containing at least two low-capability agents whose evaluator prompts instruct them to award each other high scores, and check whether DRF's reputation values and selections fail to exclude them. Alternatively, compare DRF's reputation values against known ground-truth capability (e.g., each agent's actual pass rate on the task) across many rounds; if reputation does not track the ground truth, the filtering claim collapses.","tokens_in":9776,"feed_emoji":"🤖","tokens_out":9445,"duration_ms":88824,"temperature":0.7,"pith_summary":"This paper tries to show that a multi-agent LLM system can police itself: agents rate each other's solutions, those ratings are aggregated into a reputation score for each agent, and a bandit-style rule then chooses which agents work on the next round. The claim is that this loop improves both task quality and cost compared with frameworks that fix roles in advance or stop weak agents early. The authors test the idea on code generation and logic-puzzle benchmarks and report that their framework, DRF, beats DyLAN, Reflexion, CodeT, and LLM-Debate on accuracy and average cost. The payoff, if true, is a way to build LLM teams that adapt to unknown agent reliability without human role assignment.","feed_headline":"Reputation filter boosts LLM team accuracy and cuts cost","feed_subtitle":"A rating network plus UCB selection beats fixed-role and early-stopping agent frameworks on code and reasoning tasks.","key_machinery":"The core mechanism is a three-part loop. (1) Rating network: in each round, agents generate solutions and rate each other's solutions, forming a k-layer network; an agent's round score is the reputation-weighted sum of the ratings it receives (Eq. 6), so ratings from higher-reputation agents count more. (2) Reputation iteration: scores are compared with a task threshold w0; scores at or above it increase reputation (Eq. 7), scores below decrease it (Eq. 8), separating capable agents from low-quality or malicious ones over rounds. (3) UCB selection: a bandit rule (Eq. 9) scores each agent as a weighted combination of reputation, cost, and an exploration bonus that favours rarely selected agen","core_discovery":"The paper's central claim is that an interactive rating network—where every agent both proposes a solution and evaluates the solutions of others—produces per-round scores that reveal agent capability, and that updating a reputation value from these scores, then selecting agents with a UCB (Upper Confidence Bound) rule, yields measurably better task completion and collaboration efficiency. The reputation update raises an agent's standing when its score is at or above a task threshold and lowers it otherwise; the selection rule trades off exploiting high-reputation agents against exploring untested ones, with cost included in the trade-off. In experiments on HumanEval and BigBench logic puzzle","pith_inferences":["The paper only tests lone low-quality agents; collusion between multiple low-capability agents that inflate each other's ratings is a plausible failure mode the evaluation does not cover. A direct test would seed the team with two or more low-capability agents that are prompted to rate each other highly and see whether DRF filters them.","The reliance on LLM-as-evaluator suggests that the framework's gains will be sensitive to the evaluator model's own biases; replacing the LLM evaluator with a ground-truth-based scorer (where available) would isolate the contribution of the reputation mechanism from the quality of the judge.","Because reputation is an intrinsic score carried across rounds, the authors' setting of delta=0.7 (reputation-heavy) implies low-cost agents are only preferred when reputation is roughly equal; a cost-sensitive deployment would shift delta and change the exploration-exploitation balance.","The reputation scores could be exported as a reusable credential for agents across tasks, enabling cross-task team formation if reputation transfers; the paper does not test such transfer."],"forward_implications":["If correct, DRF lets a multi-agent team operate without predefined roles or human-chosen team composition; agents self-select by demonstrated reputation.","The mechanism can identify and exclude consistently weak or malicious agents within a handful of rounds, reducing the risk of prompt-injection or low-quality contributions.","Average task cost falls because the selection rule explicitly balances reputation against cost, avoiding expensive low-quality agents.","Larger agent pools produce better results, since more high-reputation agents become available for selection.","The same framework transfers between task types: DRF outperforms baselines on both code generation and logical reasoning without task-specific tuning of roles."],"supporting_citations":[{"why":"DyLAN supplies the dynamic-team baseline and the rating-network idea DRF extends with reputation and UCB.","marker":"[10]"},{"why":"Reflexion is a comparison baseline and the source of the LLM-as-evaluator approach used in the rating network.","marker":"[13]"},{"why":"Supplies the UCB exploration bound in Eq. (9); the same marker is also used for the HumanEval dataset in Section 5.1, which the reference list does not support.","marker":"[5]"},{"why":"Prior multi-armed bandit worker-selection scheme that frames agent selection as a bandit problem.","marker":"[11]"},{"why":"LLM-Blender shows extra LLMs can rank agent contributions, motivating the rating network.","marker":"[6]"},{"why":"BigBench supplies the logical-reasoning benchmark; the same marker is also used for the LLM-Debate baseline in Section 5.3, which the reference list does not support.","marker":"[14]"},{"why":"CodeT is a comparison baseline in the HumanEval experiment; the reference list maps [20] to a truth-inference paper instead.","marker":"[20]"}],"fun_headline_variants":["Dynamic reputation filter sharpens LLM agent selection","Rating network ranks LLM agents for smarter collaboration","UCB-based reputation boosts multi-agent LLM performance","Reputation scoring improves LLM teamwork and efficiency","Interactive ratings drive better LLM agent cooperation"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Everything rests on the LLM evaluator prompts giving honest, accurate ratings of other agents' solutions, and on low-capability or malicious agents being unable to collude to inflate each other's reputations.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic reputation filter sharpens LLM agent selection","Rating network ranks LLM agents for smarter collaboration","UCB-based reputation boosts multi-agent LLM performance","Reputation scoring improves LLM teamwork and efficiency","Interactive ratings drive better LLM agent cooperation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2536,"prompt_tokens":629,"completion_tokens":1907,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":373,"completion_tokens_details":{"reasoning_tokens":1850}},"tokens_in":373,"tokens_out":1907,"duration_ms":17231,"temperature":1.0,"reasoning_tokens":1850,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:57:57.026568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same benchmarks with a team containing at least two low-capability agents whose evaluator prompts instruct them to award each other high scores, and check whether DRF's reputation values and selections fail to exclude them. Alternatively, compare DRF's reputation values against known ground-truth capability (e.g., each agent's actual pass rate on the task) across many rounds; if reputation does not track the ground truth, the filtering claim collapses.","supporting_citations":[{"cited_title":"In: Proceedings of the 24th annual conference on learning theory","cited_arxiv_id":null,"evidence_quote":"Supplies the UCB exploration bound in Eq. (9); the same marker is also used for the HumanEval dataset in Section 5.1, which the reference list does not support."},{"cited_title":"Information Sciences669, 120554 (2024)","cited_arxiv_id":null,"evidence_quote":"Prior multi-armed bandit worker-selection scheme that frames agent selection as a bandit problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CodeT is a comparison baseline in the HumanEval experiment; the reference list maps [20] to a truth-inference paper instead."}],"review_version":1}