REVIEW 4 major objections 6 minor 19 references
Whether multi-agent skill ensembles help depends on the task: big gains on open-ended math, little or harm on multiple-choice.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 21:01 UTC pith:672MQIZX
load-bearing objection Useful open-ended-vs-MCQ split on multi-agent skill mixtures, but the branded DSR/ADR/ASE stack is mostly untested and compute is confounded. the 4 major comments →
SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Task format governs whether skill-level multi-agent ensembles help. On open-ended mathematical reasoning, SKIMIX multi-agent collaboration substantially outperforms single-agent self-refinement (AIME from 40.0% to 73.3% with three agents and 76.7% with fifteen; MATH-500 from 61.0% to 72.0%). On multiple-choice benchmarks such as GPQA, self-refinement reaches 88.0% while adding agents monotonically degrades accuracy (78%, 74%, 72%). Agent-count scaling is non-monotonic, and most accuracy gains occur between round 1 and round 2.
What carries the argument
SKIMIX: parallel agents each holding a distinct skill portfolio, coordinated by a Dynamic Skill Registry (embedding-based retrieval), Anti-Dilution Routing (submodular diversity-aware portfolio selection), and iterative cross-agent share-and-refine ending in majority vote.
Load-bearing premise
The measured gains and losses are caused by skill-portfolio diversity under the proposed registry and anti-dilution machinery, rather than by extra samples, extra compute, majority vote, or the fixed hand-designed set of fifteen portfolios.
What would settle it
Run the same agent counts and refinement rounds with identical portfolios (no skill diversity) or with random portfolios, and with ADR/ASE ablated, on AIME and GPQA; if the open-ended lift and MCQ drop disappear, the skill-mixture claim fails.
If this is right
- Use single-agent self-refinement for MCQ-heavy workloads and multi-agent skill mixtures for open-ended math-style generation.
- Default agent count should be chosen empirically per task; three agents can beat five, so “more agents” is not a safe default.
- Stop after roughly two refinement rounds by default; later rounds often plateau or reverse gains.
- Majority vote leaves a coverage–accuracy gap, so better aggregation is a direct next lever for harness designers.
- Skill libraries need explicit anti-dilution, not just unlimited growth, if ensembles are to stay useful.
Where Pith is reading between the lines
- Harness routers could gate multi-agent mode on a cheap format detector (open-ended vs multiple-choice) before spending parallel compute.
- The same MCQ-versus-open-ended split may appear in tool-use or model-mixture ensembles, not only skill portfolios.
- Closing the coverage–accuracy gap with confidence-weighted or judge-based aggregation may recover many of the “lost” correct candidates without adding agents.
- If ADR and ASE were ablated and still matched Table 1, the paper’s named mechanisms would be secondary to plain parallel sampling plus refinement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes SKIMIX, a multi-agent test-time framework in which agents receive distinct skill portfolios drawn from a library and iteratively share and refine answers. It introduces three mechanisms—Dynamic Skill Registry (embedding-based retrieval), Anti-Dilution Routing (submodular relevance + log-det diversity, Eq. 3), and Adaptive Skill Evolution—and evaluates Base, Self-Refine, and SKIMIX with 3/5/15 agents on six reasoning benchmarks using DeepSeek-V3.2. The central empirical claim is that task format governs whether skill-level ensembles help: large gains on open-ended math (AIME Self-Refine 40.0% → SKIMIX-3 73.3%; MATH-500 61.0% → 72.0%) versus limited or negative gains on MCQ tasks (GPQA Self-Refine 88.0% best; multi-agent monotonically worse), with non-monotonic agent-count scaling and most accuracy lift by Round 2 (Tables 1–2).
Significance. If the task-format interaction is real and driven by complementary skill portfolios, the work supplies concrete, actionable guidance for agent harness design: prefer single-agent self-refinement on MCQ-heavy workloads and multi-agent skill diversity on open-ended reasoning, plus early stopping after ~2 rounds. The open-ended vs. MCQ divergence and the coverage–accuracy gap are useful empirical observations that are under-documented in the multi-agent scaling literature. Strengths include real API evaluation across six benchmarks, transparent per-round accuracy/coverage tables, and an explicit Limitations section. The branded mechanisms (DSR/ADR/ASE) and the causal link from skill mixture to the Table 1 pattern are not yet demonstrated, so significance currently rests on the descriptive scaling patterns rather than on a validated anti-dilution method.
major comments (4)
- [Limitations; Skill Mixture (SKIMIX); Table 1] Limitations and Experiments: The paper states that ADR and ASE “are not isolated through controlled ablation studies” and that “all experiments use a fixed library of 15 skill portfolios.” Tables 1–2 and the AIME/MATH gains are therefore produced by hand-designed fixed portfolios plus multi-agent sharing and majority vote, not by the Dynamic Skill Registry, submodular ADR (Eq. 3), or ASE. The abstract and introduction present embedding retrieval, anti-dilution routing, and adaptive evolution as core contributions that drive the results. Without at least (i) a fixed-portfolio multi-agent baseline vs. ADR-selected portfolios and (ii) a no-diversity / random-portfolio control, the causal claim that skill-mixture machinery explains the open-ended gains and MCQ losses is untested.
- [Experiments; Table 1; Agent Scaling Analysis] Experimental Settings / Table 1: Self-Refine is 1 agent × 3 rounds; SKIMIX-k is k agents × up to 3 rounds with cross-agent message passing, so SKIMIX-3 already uses roughly 3× the generations (and more tokens) before majority vote. No compute-matched or sample-matched single-agent repeated-sampling + majority-vote baseline is reported. The open-ended gains and GPQA degradation could therefore be explained by extra samples and aggregation rather than by complementary skill portfolios. A matched-budget control is load-bearing for the central “task format determines whether skill-level ensembles help” claim.
- [Table 1; Table 2; Agent Scaling Analysis] Table 1–2, AIME (n=30) and GPQA (n=50): No error bars, confidence intervals, or significance tests are reported. Base AIME accuracy is 0.0%, which is extreme and makes absolute +33-point narratives fragile. Non-monotonic agent scaling (e.g., AIME 3 > 5 agents; GPQA strictly decreasing) is highlighted as a main finding, but with small n and a single model/provider the differences may not be stable. At minimum, bootstrap CIs or repeated runs with different seeds are needed before treating non-monotonicity and the MCQ penalty as reliable design guidance.
- [Anti-Dilution Routing (ADR); Termination and Answer Selection; Table 2] Eq. (3) and Termination: The ADR objective (relevance + γ log det(I+K_P)) and the stopping rule (Δ_r ≤ λ·cost, LLM-as-Judge, min 2 rounds) are specified but never measured in the experiments—no reported values of marginal gains Δ(s|P), no archive decisions under τ_min, and no comparison of adaptive stopping vs. fixed 3 rounds on the same runs. Round-2 peak and occasional Round-3 drops are observed post hoc; they do not validate the proposed decision machinery. Either operationalize and ablate these components or narrow the claims to multi-agent skill-portfolio ensembles with fixed portfolios.
minor comments (6)
- [Abstract] Abstract vs. body inconsistency: abstract says “most improvements arise during the first refinement round”; body and Table 2 consistently show the largest jump from Round 1 to Round 2. Align wording.
- [Figures 2–6] Figures 2–6 render as corrupted Unicode glyph sequences in the manuscript text, making them unreadable. Provide clean vector plots with legible legends (method names, axes, coverage vs. accuracy).
- [Figure 8; Ablation: Refinement vs. Diversity] Figure 8 ablation bar chart labels (26.7%, 80.0%, +46.6%) do not match the prose decomposition (base 0% → self-refine 40% → SKIMIX-3 73.3%). Reconcile numbers and caption.
- [Table 1; Experimental Settings] SKIMIX-15 is omitted on MMLU-Pro, MATH-500, and BBH “due to compute constraints,” which weakens the agent-scaling and trend arrows in Table 1. Note this limitation more prominently when discussing non-monotonicity.
- [Introduction; Refinement as Message Passing] Minor typos and spacing: “SKIMIXcombinesembedding-based”, “submodularanti-dilutionrouting”, “skilldilutionbecomesarealissue”, “Ourrealexperiments(Sec.)” (empty Sec. reference). Copy-edit the PDF.
- [Related Work] Related Work: MoA and TUMIX are appropriately cited; a brief explicit comparison table (model diversity vs. tool diversity vs. skill-portfolio diversity; what is ablated) would clarify the claimed novelty boundary.
Circularity Check
No circularity: empirical multi-agent systems results on external benchmarks, not a derivation that forces outcomes from fitted or self-defined inputs.
full rationale
SKIMIX reports measured accuracies on external reasoning benchmarks (AIME, GPQA, HLE, MMLU-Pro, MATH-500, BBH) under fixed inference settings. The headline patterns—open-ended gains vs. MCQ degradation, non-monotonic agent count, Round-2 concentration of gains—are observational comparisons of Base / Self-Refine / SKIMIX-k, not quantities defined in terms of the claimed mechanisms or fitted to the reported targets. Equation (3) (submodular relevance+diversity routing) and the DSR/ASE update rules are proposed machinery; the paper does not claim a first-principles derivation that those equations equal the Table 1 deltas by construction, and Limitations explicitly notes ADR/ASE were not ablated and portfolios were a fixed library of 15. Citations (MoA, TUMIX, AutoGen, submodular maximization, etc.) are external prior art, not load-bearing self-citation uniqueness theorems. Absence of compute-matched or skill-ablated controls is a causal/attribution weakness, not circularity: nothing in the equations or evaluation protocol forces the reported accuracies from the inputs. Score 0 with empty steps is appropriate.
Axiom & Free-Parameter Ledger
free parameters (6)
- Number of agents (3 / 5 / 15) and rounds (default min 2, up to 3) =
3, 5, 15 agents; ~3 rounds
- Portfolio size k and diversity weight γ in ADR objective (Eq. 3)
- Marginal-gain archive threshold τ_min
- Embedding update step size η and alignment loss L_align (Eq. 2)
- Cost tradeoff λ in objective (Eq. 1) and stopping rule
- Decoding temperature and max tokens =
temperature 0.7, max 2048 tokens
axioms (5)
- domain assumption Diverse skill portfolios plus iterative cross-agent refinement and majority vote are a valid operationalization of skill-level test-time scaling.
- ad hoc to paper Skill dilution is well-modeled by low marginal gain in a relevance + log-det(I+K_P) submodular objective.
- domain assumption Exact-match accuracy on AIME/GPQA/HLE/MMLU-Pro/MATH-500/BBH with one backbone (DeepSeek-V3.2) supports general claims about task-format interaction.
- domain assumption Coverage (probability ≥1 agent correct) and majority vote are appropriate diagnostics/aggregators for multi-agent skill ensembles.
- standard math Greedy submodular maximization’s (1−1/e) guarantee is adequate justification for ADR in this harness.
invented entities (4)
-
Dynamic Skill Registry (DSR) with learned skill embeddings e_i
no independent evidence
-
Anti-Dilution Routing (ADR)
no independent evidence
-
Adaptive Skill Evolution (ASE)
no independent evidence
-
Pre-designed 15 skill portfolios spanning five categories
no independent evidence
read the original abstract
AI agents increasingly rely on large skill libraries, but selecting, combining, and maintaining skills remains difficult. We propose SKIMIX, a multi-agent framework in which agents with different skill portfolios collaborate through iterative refinement. SKIMIX combines embedding-based skill retrieval, submodular anti-dilution routing, and adaptive skill evolution. Across six reasoning benchmarks, multi-agent collaboration substantially improves open-ended mathematical reasoning but offers limited or negative gains on multiple-choice tasks. Agent-count scaling is non-monotonic, and most improvements arise during the first refinement round. These results show that task characteristics determine whether skill-level ensembles help and provide practical guidance for scalable agent design.
Figures
Reference graph
Works this paper leans on
-
[1]
LargeLanguageMon- keys: Scaling Inference Compute with Repeated Sam- pling
Brown,B.;Juravsky,J.;Ehrlich,R.;Clark,R.;Le,Q.V.; Ré,C.;andMirhoseini,A.2024. LargeLanguageMon- keys: Scaling Inference Compute with Repeated Sam- pling. arXiv:2407.21787
Pith/arXiv arXiv 2024
-
[2]
AgentVerse: Facilitating Multi-Agent Collaboration andExploringEmergentBehaviors.arXiv:2308.10848
Chen, W.; Su, Y.; Zuo, J.; Yang, C.; Yuan, C.; Chan, C.-M.; Yu, H.; Lu, Y.; Hung, Y.-H.; Qian, C.; Qin, Y.; Cong,X.;Xie,R.;Liu,Z.;Sun,M.;andZhou,J.2023. AgentVerse: Facilitating Multi-Agent Collaboration andExploringEmergentBehaviors.arXiv:2308.10848
Pith/arXiv arXiv 2023
-
[3]
Chen, Y.; Chen, J.; Meng, R.; Yin, J.; Li, N.; Fan, C.; Wang, C.; Pfister, T.; and Yoon, J. 2025. TUMIX: Multi-AgentTest-TimeScalingwithTool-UseMixture. arXiv:2510.01279
arXiv 2025
-
[4]
Dietterich, T. G. 2000. Ensemble methods in machine learning. InInternational workshop on multiple clas- sifier systems, 1–15. Springer
2000
-
[5]
Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Zhang, C.; Wang, J.; Wang, Z.; Yau, S. K. S.; Lin, Z.; Zhou,L.;Ran,C.;Xiao,L.;Wu,C.;andSchmidhuber, J. 2024. MetaGPT: Meta Programming for A Multi- Agent Collaborative Framework. arXiv:2308.00352
Pith/arXiv arXiv 2024
-
[6]
Krause, A.; and Golovin, D. 2014. Submodular func- tion maximization.Tractability, 3(71-104): 3
2014
-
[7]
Li, G.; Hammoud, H. A. A. K.; Itani, H.; Khizbullin, D.; and Ghanem, B. 2023. CAMEL: Communica- tiveAgentsfor"Mind"ExplorationofLargeLanguage Model Society. arXiv:2303.17760
Pith/arXiv arXiv 2023
-
[8]
F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilac- qua, M.; Petroni, F.; and Liang, P
Liu, N. F.; Lin, K.; Hewitt, J.; Paranjape, A.; Bevilac- qua, M.; Petroni, F.; and Liang, P. 2023. Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172
Pith/arXiv arXiv 2023
-
[9]
Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2023. Self-refine: Iterative refinement with self-feedback.Advances in neural information processing systems, 36: 46534–46594
2023
-
[10]
C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tür, D.; Tur, G.; and Ji, H
Qian, C.; Acikgoz, E. C.; He, Q.; Wang, H.; Chen, X.; Hakkani-Tür, D.; Tur, G.; and Ji, H. 2025. ToolRL: RewardisAllToolLearningNeeds.arXiv:2504.13958
Pith/arXiv arXiv 2025
-
[11]
Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R.Y.;Dirani,J.;Michael,J.;andBowman,S.R.2023. GPQA: A Graduate-Level Google-Proof Q&A Bench- mark. arXiv:2311.12022
Pith/arXiv arXiv 2023
-
[12]
Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q.; Hinton, G.; and Dean, J. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Mixture- of-Experts Layer. arXiv:1701.06538
Pith/arXiv arXiv 2017
-
[13]
Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2023. Reflexion: Language agents with verbal reinforcement learning.Advances in neural in- formation processing systems, 36: 8634–8652
2023
-
[14]
Srivastava, A.; Rastogi, A.; Rao, A.; Shoeb, A. A. M.; Abid, A.; Fisch, A.; Brown, A. R.; Santoro, A.; Gupta, A.; Garriga-Alonso, A.; et al. 2023. Beyond the im- itation game: Quantifying and extrapolating the capa- bilities of language models.Transactions on machine learning research
2023
-
[15]
Mixture-of-Agents Enhances LargeLan- guage Model Capabilities
Wang, J.; Wang, J.; Athiwaratkun, B.; Zhang, C.; and Zou, J.2024. Mixture-of-Agents Enhances LargeLan- guage Model Capabilities. arXiv:2406.04692
Pith/arXiv arXiv 2024
-
[16]
Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; Ren, W.; Arulraj, A.; He, X.; Jiang, Z.; et al
-
[17]
Au- togen: Enabling next-gen LLM applications via multi- agent conversations
Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang,L.;Zhang,X.;Zhang,S.;Liu,J.;etal.2024. Au- togen: Enabling next-gen LLM applications via multi- agent conversations. InFirst conference on language modeling
2024
-
[18]
Zhang, Q.; Yu, Y.; Fu, Q.; Ye, D.; et al. 2024. More agentsisallyouneed.TransactionsonMachineLearn- ing Research
2024
-
[2024]
Mmlu-pro:Amorerobustandchallengingmulti- task language understanding benchmark.Advances in Neural Information Processing Systems, 37: 95266– 95290
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.