Mixture of Complementary Agents for Robust LLM Ensemble

Fang-Yi Yu; Jie Gao; Kevin Lu; Lirong Xia; Yichi Zhang; Yuang Zhang

REVIEW 2 major objections 1 minor 1 cited by

Complementarity measured on small data guides better LLM proposer selection than accuracy or diversity alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 16:45 UTC pith:YDLIL2CJ

load-bearing objection The paper applies standard greedy feature selection to LLM proposer choice and finds some practical trade-offs, but the generalization of the complementarity scores from the labeled set is not demonstrated. the 2 major comments →

arxiv 2605.24048 v1 pith:YDLIL2CJ submitted 2026-05-21 cs.LG cs.AI

Mixture of Complementary Agents for Robust LLM Ensemble

Yichi Zhang , Kevin Lu , Yuang Zhang , Jie Gao , Lirong Xia , Fang-Yi Yu This is my paper

classification cs.LG cs.AI

keywords LLM ensembleproposer selectioncomplementaritymulti-agent collaborationgreedy selectionsummarizer modelperformance-cost trade-off

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper treats the choice of which LLMs to include as proposers in an ensemble pipeline as a combinatorial feature-selection task, where an LLM's value depends on how its outputs complement the others and the downstream summarizer. Because full combinatorial search is too expensive, the authors test a range of greedy selection procedures that estimate complementarity from responses on a small labeled set. Experiments confirm that complementarity-driven selection yields stronger ensembles and better accuracy-cost trade-offs than baselines focused only on individual model strength or response variety. A reader would care because most current multi-LLM systems still pick proposers by crude heuristics; replacing those heuristics with complementarity checks offers a concrete, low-overhead improvement path.

Core claim

Proposer selection in LLM ensembles is best solved by measuring complementarity among proposers and with the summarizer on a modest labeled set and then applying computationally feasible greedy algorithms; this approach outperforms both strongest-model and diversity-only selection while controlling total inference cost.

What carries the argument

Greedy-style selection algorithms that score complementarity from proposer responses on a small labeled set and feed the chosen subset into a fixed summarizer LLM.

Load-bearing premise

Complementarity scores computed on a small labeled set will continue to predict ensemble performance when the same proposers are used on new, unseen inputs.

What would settle it

Select a proposer subset using the complementarity method on one dataset, then measure whether that same subset still outperforms accuracy-based and diversity-based subsets when evaluated on a fresh, larger test distribution with the same summarizer.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Ensemble pipelines can achieve higher accuracy at lower total token cost by dropping redundant proposers.
The same selection logic applies to any fixed summarizer, allowing practitioners to swap in newer models without re-deriving the entire combination.
Performance gains are largest when the summarizer itself is weaker than the strongest proposers, because complementarity compensates for summarizer limitations.
The method remains practical even when the number of candidate LLMs grows, because only greedy passes over a small validation set are required.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same complementarity principle could be tested in non-LLM multi-agent systems such as tool-use planners or debate frameworks.
If complementarity is stable across domains, one could maintain a reusable library of proposer subsets instead of re-running selection for every new task.
A natural next measurement is how quickly the complementarity ranking degrades as the labeled validation set shrinks below the sizes used in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The paper applies standard greedy feature selection to LLM proposer choice and finds some practical trade-offs, but the generalization of the complementarity scores from the labeled set is not demonstrated.

read the letter

The main point here is that the authors treat choosing which LLMs to feed into a summarizer as a combinatorial selection task and test several greedy algorithms that score complementarity on a small labeled set. This is not a new algorithm but a direct transfer of feature-selection ideas to the multi-LLM setting, and they report that some of the greedy variants give better performance-cost points than picking the strongest models or maximizing diversity alone.

What the work does reasonably is lay out why pure accuracy or pairwise diversity can miss interactions with the summarizer, and it shows that a handful of computationally cheap selection rules can be run in practice. The experiments appear to compare multiple greedy heuristics and surface a few that look usable.

The soft spot is exactly the one flagged in the stress-test note. Complementarity is measured on the labeled set and then the selected subset is deployed; nothing in the abstract indicates held-out checks inside that set, cross-validation of the selection step, or explicit tests that the advantage survives when the summarizer sees new inputs. Without those, the reported trade-offs could be artifacts of the particular labeled examples rather than evidence that complementarity is a stable guiding principle. The abstract also gives no definition of the complementarity metric, no error bars, and no information on data splits or statistical tests, so the central empirical claim cannot be evaluated yet.

The paper is aimed at people who already run multi-model LLM pipelines and need a lightweight way to pick subsets. It is not advancing the theory of ensembles or feature selection. If the full manuscript contains careful validation that the selected subsets hold up on fresh data and includes the missing experimental details, it would be worth sending to referees; otherwise the contribution stays at the level of a practical heuristic note.

Referee Report

2 major / 1 minor

Summary. The manuscript reframes proposer selection for LLM ensembles (multiple proposers feeding a summarizer) as a combinatorial feature-selection problem that prioritizes complementarity among proposers and with the summarizer. It develops computationally tractable greedy-style selection algorithms that score complementarity on a small labeled set and reports experiments showing that this principle yields favorable performance-cost trade-offs compared with accuracy- or diversity-only baselines.

Significance. If the complementarity metric is shown to be stable and predictive on held-out data, the approach could supply a practical, principled alternative to existing heuristics for constructing LLM ensembles, improving robustness without exhaustive search.

major comments (2)

[Abstract and paragraph on greedy-style selection algorithms] Abstract and paragraph on greedy-style selection algorithms: the central claim that complementarity measured on a small labeled set reliably predicts summarizer performance on unseen inputs is not supported by any description of held-out validation, cross-validation of the selection procedure, or explicit checks for distribution shift between the labeled set and test inputs. This is load-bearing for the headline result on performance-cost trade-offs.
[Abstract] Abstract: no information is given on how complementarity is quantified, what statistical tests or error bars accompany the reported trade-offs, or the data splits used, so the experimental validation of the guiding principle cannot be assessed from the provided text.

minor comments (1)

[Abstract] The abstract could state the exact greedy algorithms considered and the size of the labeled set used for selection.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback emphasizing the need for clearer validation of the complementarity principle and additional experimental details. We address each major comment below.

read point-by-point responses

Referee: [Abstract and paragraph on greedy-style selection algorithms] Abstract and paragraph on greedy-style selection algorithms: the central claim that complementarity measured on a small labeled set reliably predicts summarizer performance on unseen inputs is not supported by any description of held-out validation, cross-validation of the selection procedure, or explicit checks for distribution shift between the labeled set and test inputs. This is load-bearing for the headline result on performance-cost trade-offs.

Authors: The current manuscript computes complementarity scores on a small labeled set to drive greedy selection and then evaluates the resulting ensembles on separate test inputs. However, it does not describe cross-validation of the selection procedure itself or explicit distribution-shift diagnostics. We agree these checks would strengthen the central claim and will add them (including k-fold validation of the selection step and shift analysis) in the revised manuscript. revision: yes
Referee: [Abstract] Abstract: no information is given on how complementarity is quantified, what statistical tests or error bars accompany the reported trade-offs, or the data splits used, so the experimental validation of the guiding principle cannot be assessed from the provided text.

Authors: We will revise the abstract to state that complementarity is quantified via incremental performance gain on the labeled set, to note that reported trade-offs include error bars from repeated trials, and to specify the train/test splits employed. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical validation on held-out data with no fitted quantities or self-referential derivations

full rationale

The paper presents an empirical study of greedy selection algorithms that measure complementarity on a small labeled set and evaluate performance on (implicitly separate) test inputs. No equations, fitted parameters, or derivation chain are present in the abstract or described methodology. The central claim rests on experimental outcomes rather than any reduction of a 'prediction' to its own inputs by construction. No self-citations are invoked as load-bearing uniqueness theorems, and no ansatz or renaming of known results is described. This is a standard non-circular empirical paper whose results can be falsified by new data.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The approach implicitly assumes standard greedy approximation properties hold for the LLM complementarity objective.

axioms (1)

domain assumption Greedy algorithms provide acceptable approximations to the underlying combinatorial selection problem
Paper motivates and explores greedy-style methods as computationally feasible alternatives to exact selection.

pith-pipeline@v0.9.1-grok · 5717 in / 1107 out tokens · 37011 ms · 2026-06-30T16:45:35.838297+00:00 · methodology

0 comments

read the original abstract

Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance-cost trade-offs in practice.

Figures

Figures reproduced from arXiv: 2605.24048 by Fang-Yi Yu, Jie Gao, Kevin Lu, Lirong Xia, Yichi Zhang, Yuang Zhang.

**Figure 2.** Figure 2: Distribution of the number of correct answers (bars) and summarizer accuracy [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles
cs.CL 2026-07 conditional novelty 6.0

Diversity metrics used to select LLM ensembles are largely capability proxies; after control, only a modest pairwise co-failure association with majority-vote gain remains.

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · cited by 1 Pith paper

[2]

Format your answer to include:

The single letter choice Binary-choice — Proposer Prompt You will answer a yes or no question. Format your answer to include:

work page
[3]

A concise step-by-step reasoning

work page
[4]

Your task is to analyze the reasoning of the potential solutions step by step

The yes or no answer Multi-choice — Summarizer Prompt I will give you a multiple choice question and potential solutions that may be correct or incorrect. Your task is to analyze the reasoning of the potential solutions step by step. If there are any errors, correct them and update your answer. If there are no errors, answer the question matching those so...

work page

[1] [2]

Format your answer to include:

The single letter choice Binary-choice — Proposer Prompt You will answer a yes or no question. Format your answer to include:

work page

[2] [3]

A concise step-by-step reasoning

work page

[3] [4]

Your task is to analyze the reasoning of the potential solutions step by step

The yes or no answer Multi-choice — Summarizer Prompt I will give you a multiple choice question and potential solutions that may be correct or incorrect. Your task is to analyze the reasoning of the potential solutions step by step. If there are any errors, correct them and update your answer. If there are no errors, answer the question matching those so...

work page