Pith. sign in

REVIEW 4 major objections 4 minor 2 references

Carefully designed parameter merging can consolidate specialist information-seeking agents as effectively as joint training on mixed data, at a fraction of the cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Merging separately trained information-seeking agents in weight space can rival joint data-mixing training on some knowledge-base tasks, but the paper's headline claims outrun its evidence.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A broad empirical comparison of merging vs data mixing for three agent types with a genuinely interesting LoRA failure mode, but the abstract overclaims and the missing scores plus single-run stats make the headline 'parity' claim unverifiable. the 4 major comments →

arxiv 2602.00585 v2 pith:RJL27F6X submitted 2026-01-31 cs.AI

Exploring Information Seeking Agent Consolidation

Guochen Yan , Jialong Wu , Zhengwei Tao , Bo Li , Qintong Zhang , Jiahao Xu , Haitao Mi , Yuejian Fang
show 3 more authors
Qingni Shen Wentao Zhang Zhonghai Wu
This is my paper
classification cs.AI
keywords information-seeking agentsmodel mergingdata mixingparameter-level consolidationagent consolidationRegMean++LoRAweb/document/RAG agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single unified agent for web search, document understanding, and knowledge-base retrieval can be built not only by training on a mixture of all three datasets, but also by merging the separately trained specialist models in weight space. The authors compare 26 merging operators against data-level mixing across 10 benchmarks and report that well-designed merging reaches comparable performance, is insensitive to the order in which experts were trained, and does not lose out-of-domain abilities the way mixed training does. They also extract design principles — matrix-level granularity, normalized task vectors, and informative consensus signals — and diagnose why certain popular merging strategies collapse. If true, the result matters because it offers a low-cost, privacy-friendlier route to consolidating heterogeneous agents.

Core claim

The central claim is that parameter-level consolidation, when designed with fine-grained (matrix-level) merging, normalization of task vectors, and a principled consensus strategy, can be comparable to data-level mixing for information-seeking agents, while being training-order-agnostic and better at preserving out-of-domain capabilities. The strongest evidence is on RAG, where RegMean++ outperforms data mixing on HotPotQA, AmbigQA, and Bamboogle at 30B scale, and the paper identifies the design factors behind this parity. The paper also reports that data-level mixing is catastrophically sensitive to training order (a Web-to-Doc-to-RAG sequence drops web performance by 74%) and that subspace

What carries the argument

The central object is the parameter-merging operator applied to task vectors — the difference between each fine-tuned expert and the shared base model. The paper organizes these operators by granularity (model, layer, matrix), consensus strategy (sign consistency, subspace filtering, activation matching), and data dependence, then evaluates them under full fine-tuning and LoRA. The load-bearing design choices are matrix-level merging and consensus based on activation-space alignment (as in RegMean++'s closed-form regression-mean solution), together with normalization of update magnitudes before merging.

Load-bearing premise

The paper's method rankings rest on single-run accuracy scores computed on small sampled subsets (200 BrowseComp, 100 BrowseComp-zh, 103 GAIA text) with no reported variance or significance testing, so evaluation noise could flip the relative standing of methods that differ by a few points.

What would settle it

Re-run the top methods (RegMean++, data mixing, Average, TIES) on 10 random subsets of BrowseComp and BrowseComp-zh of the same size and compute confidence intervals; if RegMean++ does not consistently beat data mixing on RAG across subsets and seeds, the parity claim and design principles lose support.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Merging specialists in weight space can reach data-mixing-level performance at a fraction of training cost, with RegMean++ surpassing data mixing on RAG benchmarks at 30B scale.
  • Unlike data mixing, which drops 74% web and 59% document performance when trained Web-to-Doc-to-RAG sequentially, parameter merging is order-agnostic.
  • Parameter merging preserves out-of-domain capabilities that data mixing universally forgets, per the paper's finding (ii).
  • Effective merging follows design principles: matrix-level granularity, task-vector normalization, and informative consensus strategies; heuristic sign-consistency methods are fragile.
  • Subspace-based merging methods fail on LoRA-trained experts due to near-orthogonal low-rank updates, so consolidation design must be adaptive to the training paradigm.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the parity result holds broadly, merging could let developers consolidate agents trained on private or regulated data without ever pooling the trajectory data.
  • The order-agnostic property suggests merging as a natural continual-learning or model-update mechanism when new specialist agents are added over time.
  • The gap between data-free heuristics and RegMean++'s small unlabeled-sample requirement hints that a 'few-shot calibration' middle ground may dominate future practice.
  • The LoRA orthogonality diagnosis implies a testable extension: merging operators that explicitly preserve subspace orthogonality should recover LoRA merging performance; conversely, random LoRA seeds with engineered shared subspaces should make current subspace methods viable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an empirical comparison of two strategies for consolidating information-seeking agents specialized in web search, document understanding, and knowledge-base retrieval: data-level consolidation (training on a mixture of trajectories) and parameter-level merging (combining independently trained experts). The authors evaluate 20 merging methods on 8 benchmarks using Qwen3-30B-A3B-Think and Qwen3-4B-Think backbones, including full fine-tuning and LoRA, and analyze tool-call behaviors. The body's conclusions are hedged: data-level mixing is a strong baseline, while a few parameter-level methods are comparable on some RAG metrics. The abstract, however, claims 26 methods, introduces a geometric Composite Score and an Imbalance Score, and asserts parity with data mixing, preservation of out-of-domain capabilities, and cost advantages, none of which are operationalized in the body.

Significance. If the reported findings were fully supported, the study would provide a useful design guide for consolidating heterogeneous information-seeking agents without joint retraining, which is practically valuable given the cost of training such agents. Strengths include the systematic coverage of 20 merging methods, the use of two backbone scales and both full-parameter and LoRA training, the behavioral analysis of tool-use patterns, and the distillation of concrete design principles. However, the manuscript in its current form does not deliver the abstract's headline claims: the promised composite scores are absent, the number of evaluated methods is inconsistent, and the empirical inferences rest on single-run, small-subset evaluations without uncertainty quantification. These issues must be resolved before the practical conclusions can be relied upon.

major comments (4)
  1. [Abstract vs. §4/Table 2] The abstract promises a 'geometric Composite Score' and an 'Imbalance Score' for cross-benchmark comparison, but neither is defined, computed, or referenced anywhere in the body. The abstract also states that 26 parameter-level methods were evaluated, while the body (Table 2 and the contributions bullet) evaluates 20. Because the central claim that parameter-level merging attains 'parity with data mixing' depends on an undefined aggregation across heterogeneous benchmarks, that claim cannot be evaluated from the manuscript as written. Please either define and report these scores with explicit formulas and values, or revise the abstract to match the body's hedged conclusions.
  2. [§4, Tables 3–5] All performance numbers are single-run; no seeds, variance, or significance tests are reported. The text notes that BrowseComp and BrowseComp-zh are randomly subsampled (200 and 100 instances) and GAIA is a 103-instance subset. Many rank-relevant differences are a few points (e.g., Table 3: GAIA 64.08 vs 64.07; BC 28.00 vs 23.00; Bamboogle EM 53.10 vs 52.80). Without uncertainty quantification, the method rankings and the design principles drawn from them (matrix-level granularity, consensus strategy, subspace methods) are not reliably established. Please report multiple seeds or error bars, or at least a significance/effect-size analysis, and temper the conclusions accordingly.
  3. [§5.1, RQ I; abstract claim (ii)] The abstract states that parameter-level merging 'structurally preserves out-of-domain capabilities that data mixing universally forgets.' No table or section in the body presents evidence of forgetting by data mixing. Finding ❶ in §5.1 discusses RAG as out-of-distribution and notes that consolidation methods rarely surpass the dedicated RAG agent, which is a different claim. Please either add direct evidence of data mixing forgetting or remove/qualify this abstract claim.
  4. [Abstract claim (i) / §5.1] The abstract claims parity 'at a fraction of its training cost,' but no training or merging cost is quantified anywhere in the manuscript. The paper only asserts that parameter-level merging avoids joint retraining. Without actual compute/cost measurements, the efficiency advantage is not demonstrated. Please include cost comparisons or substantially qualify the claim.
minor comments (4)
  1. [Abstract / Table 2] The abstract says 26 methods and 10 benchmarks; the body says 20 methods and lists 8 benchmarks (GAIA, BrowseComp, BrowseComp-zh, MMLongBenchDoc, DocBench, HotPotQA, AmbigQA, Bamboogle). Please reconcile these numbers.
  2. [Tables 3–5 captions] The caption 'The number indicates the number of cases in which the method outperforms the expert agent' appears inconsistent with the table cells, which appear to be accuracy/F1 scores. Please clarify or correct the caption.
  3. [References] Wei et al. 2025a and 2025b, and Wu et al. 2025a and 2025b, are duplicate entries for the same papers. Please merge or disambiguate.
  4. [§4] Minor typo: '20 instinct representative parameter-level consolidation methods' should read '20 representative' or '20 instances'.

Circularity Check

0 steps flagged

Empirical comparison with external benchmarks; no derived quantity reduces to its fit or to a self-citation.

full rationale

This paper is an empirical benchmarking study rather than a derivation, so the main circularity patterns do not apply. The central comparison (data mixing vs. parameter merging) is measured on external benchmarks (GAIA, BrowseComp, HotPotQA, etc.) using published merging operators, and no parameter is fitted to a target metric and then reported as a prediction. The self-citations to Webdancer/Docdancer/Ragshaper are used only to construct the three expert agents; the paper's conclusions are not inferred from those papers' claims, so they are constructional rather than load-bearing. The order-agnostic statement is a tautological property of the symmetric merge definition (Eq. 4), and the abstract's promised 'geometric Composite Score and an Imbalance Score' never appear in the body, leaving the headline 'parity' claim undefined. Both are evidential/definitional weaknesses, not circular reductions: the tables still contain independent measurements, and no claimed result reduces to its own input by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The paper introduces no physical entities; its invented constructs are two promised evaluation metrics that do not appear in the body. The free parameters are mostly unstated experimental choices (merge coefficients, LoRA rank, evaluation subset sizes) that the findings depend on.

free parameters (3)
  • merge coefficients for Average/SLERP (α_web, α_doc, α_rag)
    Eq. 4 defines convex-combination averaging but the paper never reports which α values are used; the parity results depend on these choices.
  • LoRA rank = 8
    Appendix B sets LoRA rank to 8 for all linear layers; the LoRA-specific failure analysis (TSV/WUDI collapse) is specific to this rank choice.
  • evaluation subset sizes = 200 (BC), 100 (BC-zh), 103 (GAIA)
    Subsampled due to cost; the fine-grained rankings of methods are computed on these small subsets without variance.
axioms (4)
  • domain assumption ReAct-style trajectory format is the correct common abstraction for web, doc, and RAG agents
    §3.1 assumes all three environments reduce to thought/action/observation loops; tool-use statistics are cited as support but the representativeness is taken as given.
  • domain assumption Benchmark answers via LLM-as-Judge for MMBD and DocB are reliable
    Appendix C.2 reports doc accuracy via LLM-as-Judge with no human calibration or inter-annotator check.
  • domain assumption Training trajectories (Dweb, Ddoc, Drag) do not leak evaluation benchmarks
    Training data follows the authors' prior synthesis pipelines (Webdancer/Docdancer/Ragshaper); no statement confirms test instances are excluded, so contamination risk is unaddressed.
  • domain assumption GAIA/BrowseComp subsets of 100-200 instances are sufficient to rank methods
    §4 samples subsets for cost; conclusions about orderings among methods rely on these small samples without confidence intervals.
invented entities (2)
  • Geometric Composite Score no independent evidence
    purpose: Abstract claims it aggregates heterogeneous benchmark performance for cross-benchmark comparison
    Announced in the abstract ('we introduce a geometric Composite Score and an Imbalance Score') but never defined, derived, or applied anywhere in the body; no formula, no table, no release.
  • Imbalance Score no independent evidence
    purpose: Abstract claims it measures task skew across benchmarks
    Same as Geometric Composite Score: promised, absent.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Information Seeking Agent Consolidation." pith.science (2026). https://pith.science/paper/RJL27F6X

@misc{pith2026260200585,
  author       = {Pith},
  title        = {Pith review of: Exploring Information Seeking Agent Consolidation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RJL27F6X}},
  note         = {Machine review of arXiv:2602.00585}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Information-seeking agents have emerged as a powerful paradigm for knowledge-intensive tasks, yet today's systems remain specialized for the open web, documents, or local knowledge bases, hindering scalable and cross-domain deployment. We present the first systematic empirical study of consolidating these information-seeking agents into a single foundation agentic model. We compare two paradigms -- \emph{data-level mixing}, which trains a unified model on a mixture of datasets, and \emph{parameter-level merging}, which merges independently trained experts in parameter space -- across 3 training scenarios, evaluating \textbf{26} representative parameter-level methods on \textbf{10} benchmarks. To compare across heterogeneous benchmarks, we introduce a geometric Composite Score and an Imbalance Score that describe overall performance and task skew. Our analysis shows that (i) well-designed parameter-level merging attains parity with data mixing at a fraction of its training cost and is order-agnostic; (ii) parameter-level merging structurally preserves out-of-domain capabilities that data mixing universally forgets; and (iii) cross-scenario stability is strongly tied to consolidation quality. We distil our observations into a method-selection guide and design principles for next-generation merging operators.

Figures

Figures reproduced from arXiv: 2602.00585 by Bo Li, Guochen Yan, Haitao Mi, Jiahao Xu, Jialong Wu, Qingni Shen, Qintong Zhang, Wentao Zhang, Yuejian Fang, Zhengwei Tao, Zhonghai Wu.

Figure 1
Figure 1. Figure 1: Comparison of three information-seeking agent consolidation paradigms. (a) Single-task training, where separate agents are independently trained for local knowledge-base retrieval Drag, document understanding Ddoc, and open-web search Dweb, in their respective environments. (b) Data-level consolidation, which unifies heterogeneous agent trajectories into a single training set Dall and learns a single model… view at source ↗
Figure 2
Figure 2. Figure 2: Average tool usage frequency and average answer length across different information-seeking settings in training data. first train a set of models independently, each specialized for a particular environment, and then consolidate them by merging their parameters to obtain a generalized model: θ (merge) = merge(θweb, θdoc, θrag), where merge(·) denotes a generic parameter merging oper￾ator. One straightforw… view at source ↗
Figure 3
Figure 3. Figure 3: Differences of information-seeking behavior between consolidation methods and the expert agent across benchmarks on Qwen3-30B-A3B-think. We select the top-performing parameter￾level consolidation method, RegMean++, as the representative. Re￾sults are reported across multiple information-seeking categories, with detailed definitions of each category provided in Appendix G. specific principal components that… view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise L2 norm of parameter updates of expert agents for Web, Doc, and RAG agents across model depth. which significantly limits their applicability in resource￾constrained or privacy-sensitive settings. ❺ AdaMerging shows unstable performance (as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training loss curves and benchmark performance during training of Qwen3-30B-A3B-Think. the agent’s resilience and its ability to perform multi-step retrieval when direct search fails. D. Agents D.1. Web Agent The web agent employs two types of tools, following previ￾ous work (Wu et al., 2025a; Tao et al., 2025): Search and Visit: • Search is used to retrieve information via the Google search engine. Its in… view at source ↗
Figure 6
Figure 6. Figure 6: Training loss curves and benchmark performance during training of Qwen3-4B-Think. and documents indexed in the KB. It returns documents whose similarity scores exceed a threshold τ , while en￾suring that the number of returned documents does not exceed k. D.4. Tool Schema This section details the tool schemas provided to the agent. The specific JSON structures defining the tools for the Web agent, Doc agen… view at source ↗
Figure 7
Figure 7. Figure 7: Training loss curves and benchmark performance during training of Qwen3-30B-A3B-Think with LoRA. 0 2 4 6 8 10 12 14 0 2 6 8 10 0 2 4 6 8 10 12 14 16 0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50 0 5 10 15 20 25 30 35 40 45 50 Tool Usage Distribution Single Task Data-level Consolidation Parameter-level Consolidation [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Tool call distributions of different consolidation strategies across Web benchmarks. 0 5 10 15 20 25 30 35 0 5 10 15 20 30 35 40 25 0 5 10 15 20 30 35 40 25 0 2 4 6 8 10 12 14 16 18 20 22 24 25 >25 0 2 4 6 8 10 12 14 16 18 20 22 24 25 >25 0 2 4 6 8 10 12 14 16 18 20 22 24 25 >25 Tool Usage Distribution Single Task Data-level Consolidation Parameter-level Consolidation [PITH_FULL_IMAGE:figures/full_fig_p01… view at source ↗
Figure 9
Figure 9. Figure 9: Tool call distributions of different consolidation strategies across Doc benchmarks. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Tool call distributions of different consolidation strategies across RAG benchmarks. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Tool schema for web agent: Search and Visit. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Tool schema for doc agent: Search and Read. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Tool schema for RAG agent: dense semantic retrieval over a vectorized knowledge base. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages · 1 internal anchor

  1. [2020]

    type": "function

    and computes similarity scores between the query 3https://github.com/facebookresearch/DPR 14 Exploring Information Seeking Agent Consolidation Web RagDoc 0501001502002503001.0 1.2 1.4 1.6 0.81.01.21.41.6 0.81.01.21.41.6 00.81.01.21.41.6 Data-level Consolidate Loss Step GAIABCBC-zh 31.073.002.0052.439.0012.0050.4914.0015.00 MMBD(LasJ)DocB(LasJ) HP(F1)AmBig...

    arXiv 2065

  2. [2025]

    One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging

    doi: 10.48550/ARXIV .2508.06163. URLhttps: //doi.org/10.48550/arXiv.2508.06163. Ma, Y ., Zang, Y ., Chen, L., Chen, M., Jiao, Y ., Li, X., Lu, X., Liu, Z., Ma, Y ., Dong, X., et al. Mmlongbench- doc: Benchmarking long-context document understand- ing with visualizations.Advances in Neural Information Processing Systems, 37:95963–96010, 2024. Maiti, S., Bu...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.06163 2024

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.